Production LLM System Design Architecture: Model Routing, Token Caching, and Infrastructure Sizing ($120k–$30…

The Four-Layer Production LLM Architecture

Most production LLM failures stem from treating an LLM as a simple REST API endpoint. When traffic spikes or upstream providers hit rate limits, latency degrades, costs skyrocket, and applications freeze. A resilient production architecture isolates business logic from model execution across four distinct layers.

## Simplified Gateway Route Configuration (LiteLLM / Envoy pattern)
model_list:
  - model_name: tier1-reasoning
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      tpm: 80000
      rpm: 1000
  - model_name: tier2-fast
    litellm_params:
      model: openai/gpt-4o-mini
      tpm: 200000
      rpm: 3000
  - model_name: tier3-local
    litellm_params:
      model: openai/meta-llama-3.1-8b-instruct
      api_base: http://vllm-cluster.internal:8000/v1

1. Ingestion and Gateway Layer

The entry point handles rate limiting, authentication, payload validation, and request queuing. It acts as an abstraction proxy so client applications interact with standard schemas rather than vendor-specific SDKs.

2. Context and Memory Layer

This layer builds the payload injected into the model prompt. It retrieves context from vector databases (Qdrant, pgvector), fetches real-time entity state from Redis, and truncates historical messages to stay within token windows.

3. Routing and Orchestration Layer

The routing tier inspects the incoming payload complexity, user entitlement level, and strict SLA requirements. It routes the prompt to the cheapest model capable of completing the job, applying semantic caching before hitting downstream APIs.

4. Execution and Inference Layer

The execution target: either commercial APIs (OpenAI, Anthropic) or self-hosted inference clusters (vLLM, TensorRT-LLM, Ray Serve) running on dedicated GPU infrastructure.

Dynamic Model Routing: Balancing Latency and Cost

Routing every request to Claude 3.5 Sonnet or GPT-4o burns budget needlessly. Roughly 60% of user queries in standard enterprise SaaS applications—such as data extraction, text reformatting, or simple classification—can be handled by smaller models like Llama 3.1 8B or GPT-4o-mini at a fraction of the cost.

A dynamic model router inspects query complexity using low-latency heuristics:

• Token Length and Structure: Short, highly constrained prompts (JSON schemas) get routed to fast, small models.
• Semantic Intent Classification: A lightweight classifier (such as a BERT-mini model or BGE-small embedder running on CPU) categorizes incoming tasks into simple retrieval, summarization, or complex multi-step reasoning.
• Fallback Escalation: If a tier-3 model fails output validation (such as returning invalid JSON), the router automatically re-tries the request against a tier-1 model.

Teams deploying our custom LLM development services frequently implement this tiered pattern to lower monthly API spend by 40% to 65% without degrading output quality.

Token Caching Strategies: Exact vs. Semantic

Inference is expensive because transformer attention scales quadratically with context length. Effective caching avoids redundant compute entirely. Production designs use a two-tier caching pattern:

## Semantic Cache Check Flow (Redis + Embedding Vector Search)
import redis
import numpy as np

def get_cached_response(prompt_embedding: list[float], threshold: float = 0.96):
    r = redis.Redis(host='redis.internal', port=6379)
## Query Redis VSS index for nearest neighbor
    query = (
        f"*=>[KNN 1 @vector $vec AS score]"
    )
    results = r.ft("cache_index").search(
        query, 
        query_params={"vec": np.array(prompt_embedding, dtype=np.float32).tobytes()}
    )
    if results.docs and float(results.docs[0].score) >= threshold:
        return results.docs[0].response
    return None

Exact Match Caching

Hashes the exact normalized prompt string (SHA-256) along with system instructions and context IDs.

• Storage: Redis or Memcached.
• Latency: < 5ms lookup.
• Hit Rate: 10%–20% in predictable structured applications.

Semantic Caching

Embeds the incoming query and performs a fast vector similarity search against past queries. If cosine similarity exceeds a defined threshold (typically 0.95–0.98), the cached response is served.

• Storage: Qdrant, Milvus, or Redis Vector Search Index.
• Latency: 15ms–30ms lookup.
• Hit Rate: 25%–45% in open-ended conversational interfaces.

Upstream Prompt Caching

Modern API providers (Anthropic, OpenAI, DeepSeek) support prompt prefix caching. By placing static context—such as system instructions, document embeddings, and schema definitions—at the front of the prompt payload, repeated calls reuse cached KV matrices directly on GPU memory, cutting input token costs by up to 90% and reducing TTFT by 80%.

Infrastructure Sizing and Cost Modeling ($120k–$300k Projects)

When evaluating self-hosted vLLM clusters on AWS or GCP against commercial APIs, infrastructure sizing determines whether self-hosting saves money or drains budget.

Engineering managers must evaluate throughput requirements based on peak tokens per second (TPS).

Self-Hosted Capacity Formula:
Required GPUs = (Peak Users * Avg Output Tokens/Sec) / (Tokens/Sec per GPU at SLA Target)

For an enterprise application processing 5 million queries per month with an average prompt size of 1,500 input tokens and 300 output tokens:

Commercial API Hosting Model

• Input Tokens: 7.5 Billion / month
• Output Tokens: 1.5 Billion / month
• Blended API Cost (GPT-4o-mini / Claude 3 Haiku mix): ~$2,100 / month
• Blended API Cost (Claude 3.5 Sonnet heavy mix): ~$45,000 / month

Self-Hosted vLLM Infrastructure Model (Llama 3.1 70B on 4x NVIDIA A100 80GB)

• Instance Cost (AWS p4d.24xlarge reserved or specialized GPU clouds like Lambda/RunPod): ~$4,800 – $7,200 / month
• Kubernetes/Karpenter Cluster Overhead + Networking: ~$1,200 / month
• Total Infrastructure Cost: ~$6,000 – $8,400 / month

The Engineering Reality: Self-hosting only pays off if query volume exceeds 10 million requests per month, or if strict data privacy rules prohibit sending payload text across external vendor boundaries. For engagements within the $120k to $300k build budget range, we typically recommend a hybrid architecture: API-first routing with targeted open-source hosting for high-throughput, narrow-task pipelines. Teams building out these architectures rely on our AI development services to size compute footprint correctly from day one.

Latency Budgeting, Fallbacks, and Degradation Strategies

In an LLM system design, downstream services will fail, rate limits will be breached, and provider latency will spike. Your system needs a deterministic latency budget:

Soft Timeouts and Degradation Rules

1. TTFT Breached (> 1,200ms): Cancel the request to the primary provider (e.g., Anthropic) and fall back immediately to an ultra-fast secondary stream (e.g., Groq-hosted Llama or GPT-4o-mini).
2. Context Compression on Timeout: If RAG retrieval exceeds its 200ms budget, bypass dense vector search, grab top-K exact keyword results from Elasticsearch/pgvector, compress the context window, and fire the LLM call.
3. Graceful UI Streaming: Use Server-Sent Events (SSE) or WebSockets to stream tokens to the user UI as they arrive. Streaming masks total completion time by lowering perceived latency to the TTFT metric.

Observability, Guardrails, and Evaluation Pipelines

Standard APM tools like Datadog or New Relic fall short when tracking non-deterministic LLM pipelines. Production systems require LLM-native telemetry tracking four core dimensions:

• Token Economics: Real-time tracking of input, output, and cached token counts per tenant, user ID, and feature endpoint.
• Latencies Breakdown: Dissecting queue time, TTFT, and generation speed (tokens/second) across every model target.
• Input/Output Guardrails: Running light checking engines (NeMo Guardrails, Llama Guard) asynchronously or in-line to prevent prompt injection, PII leakage, and off-topic queries.
• Continuous Evaluation (LLM-as-a-Judge): Sampling 2% to 5% of completed production traces, storing them in data warehouses (Snowflake, BigQuery), and scoring them asynchronously with evaluation frameworks (Ragas, DeepEval) to detect output drift over time.

Additionally, understanding how web crawlers interact with public-facing AI engines is critical for platform visibility. We track these AI bot patterns directly in our AI Answer-Engine Crawl Index.

What This Means for Your Team

Building a production-ready LLM architecture isn't about calling an API; it's about building resilient software around non-deterministic components.

• Decouple immediately: Abstract model APIs behind a gateway pattern (LiteLLM, Envoy, or custom proxy) on day one.
• Control costs with dynamic routing: Send 70% of routine traffic to low-cost models or cached endpoints, reserving tier-1 models for complex tasks.
• Budget for latency: Enforce strict TTFT timeouts, use Server-Sent Events for streaming responses, and implement automatic provider fallbacks.
• Size infrastructure carefully: Avoid prematurely self-hosting GPUs until monthly query volume justifies the operational complexity and node reservation costs.

If you are planning an AI engineering initiative in the $120k to $300k range and need a battle-tested architecture designed and delivered by senior engineers, contact us to review your system specs.