Enterprise LLM Gateways: Comparing LiteLLM vs Portkey vs Custom Proxies for Rate Limiting and Audit Logs

An enterprise LLM gateway routes, monitors, and standardizes traffic across disparate AI providers through a unified API. LiteLLM is the industry default for self-hosted, Python-native teams needing raw proxying with zero data leakage. Portkey excels when product teams need managed observability, guardrails, and fine-grained prompt logging out of the box. Custom proxies rarely make financial sense unless your infrastructure mandates bespoke protocol translation or sub-millisecond gRPC routing.

Why engineering teams deploy LLM gateways

Most engineering teams realize they need an LLM gateway after a billing incident or a compliance audit.

Without a centralized proxy layer, individual product squads embed raw OpenAI, Anthropic, or Bedrock API keys directly into microservices. That pattern creates three predictable engineering failures:

1. Unbounded spend and thundering herds. A runaway recursive prompt loop or a sudden spike in batch evaluation runs trips provider-tier rate limits, knocking your entire production application offline.
2. Zero compliance observability. Security asks which employees or services sent PII to external endpoints last Tuesday, and your logs contain only raw HTTP 200 counts from an API gateway without request payloads.
3. Provider lock-in and brittle failover. If OpenAI experiences a global degradation, engineering must execute emergency deploys across four services to swap client SDKs to Claude 3.5 Sonnet on AWS Bedrock.

An LLM gateway acts as a reverse proxy sitting between your application code and upstream model providers. It translates diverse vendor formats into a standardized OpenAI-compatible schema, enforces tenancy-based rate limits via Redis, tracks cost per team or API key, and routes requests across fallbacks when an upstream API returns a 500 error.

How do LiteLLM, Portkey, and custom proxies compare?

LiteLLM provides the cleanest raw self-hosted proxying with transparent pricing, while Portkey delivers turnkey governance and request tracing at the expense of vendor dependency. Building a custom proxy from scratch is an expensive engineering trap that trades low infrastructure costs for endless SDK maintenance.

The following matrix reflects production deployments serving between 2 million and 80 million tokens daily.

Teams modernizing core pipelines through targeted AI development services typically start with a self-hosted instance of LiteLLM to eliminate vendor risk while maintaining complete network control inside their VPC.

LiteLLM deep dive: Architectural strengths and operational trade-offs

LiteLLM's core value proposition is straightforward: it exposes an OpenAI-compatible endpoint that translates your requests on the fly to Anthropic, Bedrock, Vertex AI, Azure OpenAI, or local vLLM instances.

Deploying LiteLLM requires a containerized runtime (ECS, EKS, or Cloud Run) backed by a PostgreSQL database for team/key metadata and a Redis instance for global rate limiting.

Production configuration

A minimal production config.yaml for LiteLLM handles fallback routing, virtual keys, and database-backed audit logging without requiring application code changes:

model_list:
  - model_name: production-inference
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 2000
      tpm: 160000
  - model_name: production-inference
    litellm_params:
      model: azure/gpt-4o
      api_base: https://corp-east-01.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY
      rpm: 4000
      tpm: 320000

router_settings:
  routing_strategy: usage-based-routing
  redis_host: os.environ/REDIS_HOST
  redis_port: 6379
  redis_password: os.environ/REDIS_PASSWORD
  num_retries: 3
  timeout: 45

general_settings:
  database_url: os.environ/DATABASE_URL
  store_model_in_db: True

Where LiteLLM breaks down

LiteLLM's velocity is also its weakness. The maintainers merge pull requests rapidly to keep up with daily AI model releases. Occasionally, upstream changes introduce minor regressions in token counting or edge-case streaming chunks over SSE (Server-Sent Events).

If you pick LiteLLM, lock your container image tags, pin your dependencies, and run regression tests against streaming endpoints before promoting updates to production clusters.

Portkey deep dive: Turnkey observability and enterprise guardrails

Portkey prioritizes governance and developer experience over bare-bones protocol translation. While it offers an open-source gateway core, its commercial appeal lies in its hosted control plane and deep tracing primitives.

If your organization has legal teams mandating full prompt/response audits, PII masking before data touches third-party models, and visual debugging for product managers, Portkey's control plane solves those problems on day one.

Key capabilities

• Virtual Keys and Budgets: You can assign virtual API keys to specific business units, attach strict monthly dollar caps, and configure automatic email alerts when a team hits 80% of its allocation.
• Granular Guardrails: Portkey lets engineers intercept incoming prompts and evaluate them against pre-flight checks (regex, toxicity models, semantic vector checks) before routing to the provider.
• Visual Tracing: Every execution graph, prompt template variation, and fallback event is indexed with latency, token count, and exact dollar cost down to four decimal places.

The operational reality

Portkey is ideal when you want to avoid dedicating half a staff engineer's time to maintaining an internal developer portal for LLM keys. However, routing payloads through a multi-tenant cloud control plane introduces third-party data processing agreements (DPAs) and vendor lock-in.

Portkey offers an enterprise on-premise installation, but the infrastructure footprint is significantly heavier than LiteLLM's lightweight proxy containers.

Why building a custom proxy is usually a mistake

Almost every senior engineer who has designed high-throughput distributed systems looks at an LLM gateway and says: "It's just an HTTP reverse proxy with a Redis counter. We can write this in Go over two sprints."

That calculus ignores the ongoing maintenance tax of upstream model APIs.

Engineering Hours Spent on In-House LLM Proxy:

Sprint 1-2:   [████████████████] Basic proxy + Redis token bucket (Works)
Month 2:      [████████] Handling SSE streaming chunk discrepancies
Month 3:      [████████████] Anthropic releases breaking tool-call format
Month 4:      [████████████████] Building cost database + token counters
Month 6:      [████████████████████] Updating Bedrock IAM role parsing

To build a custom gateway, your team must maintain:

1. Schema normalization: Anthropic, OpenAI, Google Gemini, and AWS Bedrock handle system prompts, function calling, tool use, and multimodal base64 payloads differently. You will write endless marshaling logic.
2. Accurate token calculation: Upstream tokenizers change with each model family. Tiktoken does not count tokens the same way Claude's tokenizer does. If your proxy counts tokens incorrectly, your rate-limiting windows and cost-allocation ledgers will drift out of sync with actual vendor invoices.
3. Complex streaming responses: Normalizing streaming Server-Sent Events while calculating per-chunk usage metrics without inflating latency requires battle-tested concurrency code.

Unless your infrastructure must handle sub-5-millisecond p99 latencies within a strictly regulated on-premises data center that prohibits external runtime code, do not build this from scratch. Use battle-tested systems designed during dedicated LLM development services rather than reinventing provider routing.

How much does an enterprise LLM gateway cost?

Deploying a production LLM gateway costs between $15,000 and $90,000 in Year 1 for commercial tools or open-source infrastructure, compared to $120,000 to $250,000 if built from scratch. Open-source deployments trade SaaS subscription fees for cloud hosting and internal maintenance labor.

The financial breakdown depends heavily on your team's throughput and compliance requirements:

• LiteLLM Self-Hosted: Free open-source software. An enterprise license costs between $20,000 and $50,000 annually if you require SSO, audit vaulting, and SLA support. AWS infrastructure (EKS cluster, Redis instance, managed RDS PostgreSQL) adds $400 to $1,200 per month depending on redundancy.
• Portkey: Free tier covers dev workflows. Enterprise plans generally price on a platform fee plus per-million-requests or per-seat billing, running $1,500 to $6,000+ per month for enterprise volume.
• Custom Proxy: Low infrastructure cost ($200 to $500 monthly for lightweight Go instances), but consumes 0.5 to 1.0 dedicated full-time engineer equivalents (FTE) across initial build and quarterly upstream API updates. At US engineering rates, building in-house costs at least $120,000 in diverted developer capacity.

How long does an LLM gateway migration take?

A production gateway migration takes 2 to 4 weeks from initial container deployment to complete production traffic cutover across all services. The gateway deployment itself takes 48 hours; the remainder is spent updating internal application configurations and auditing log pipelines.

A standard rollout sequence follows a strict zero-downtime phased migration:

1. Week 1 (Infrastructure & Key Provisioning): Stand up the gateway in a staging VPC. Configure PostgreSQL for user tables and Redis for rate-limiting tiers. Provision master provider keys and generate virtual team keys.
2. Week 2 (Shadow Routing & Latency Auditing): Route non-critical background jobs or batch evaluations through the proxy. Instrument Datadog or OpenTelemetry traces to confirm the gateway adds no more than 15ms to 25ms of p95 latency overhead.
3. Week 3 (Core Service Redirection): Point production microservice base URLs to the internal gateway URL. Set client timeouts to 60 seconds to accommodate slow reasoning models. Configure automated upstream fallbacks (e.g., primary: Claude 3.5 Sonnet, secondary: GPT-4o).
4. Week 4 (Enforcement & Deprecation): Revoke direct API keys at the provider level. All egress traffic to api.openai.com and api.anthropic.com not originating from the gateway IP block is dropped at the firewall.

How should you handle audit logs and PII compliance?

To maintain SOC 2, HIPAA, or ISO 27001 compliance, an LLM gateway must capture request metadata without storing unencrypted, sensitive customer data in plain-text logs.

Engineers face a structural tension here: debugging agent hallucinations requires seeing the exact prompt that broke, but compliance teams prohibit storing PII in searchable operational databases.

Implement a dual-stream audit architecture to solve this:

LiteLLM integrates directly with S3 buckets and OpenTelemetry collectors, allowing you to route stripped traces to Datadog while dumping the encrypted raw payload directly to an access-controlled S3 bucket. Portkey provides native guardrails that can mask credit card numbers, social security numbers, and email addresses inline before the request leaves your cloud perimeter.

What this means for your team

Deploying an LLM gateway is no longer an optional architectural step once you have more than two services consuming external AI models.

If you are a remote-first engineering organization with strict VPC isolation mandates and existing Kubernetes talent, deploy LiteLLM. It preserves your capital, keeps data in your network, and immediately stops runaway API spend through Redis-backed rate limiting.

If you have a non-technical product team that needs to iterate on prompt templates, adjust fallback logic visually, and review compliance traces without opening JIRA tickets, purchase Portkey.

Avoid building an in-house proxy unless your technical requirements genuinely exist outside standard HTTP and SSE transport specifications. The engineering cost of keeping pace with daily model releases far outstrips the operational overhead of running established gateway tooling.

To audit your current AI infrastructure, design a resilient model routing layer, or evaluate enterprise gateway deployments, talk with our engineering team.