Best LLM gateways in 2026: 30+ AI gateways compared on cost control

Enterprises spent $37 billion on generative AI in 2025, a 3.2x jump in one year, per Menlo Ventures. Almost none of that traffic passes through anything a finance team can see. Every AI call your company makes crosses exactly one convenient chokepoint, and most teams haven’t claimed it yet.

This guide compares more than 30 gateways on the axis other roundups skip: which ones actually let you see, allocate, and control what AI costs you. Because a gateway that routes brilliantly but can’t tell finance who spent what is doing half its job.

What is an LLM gateway, and is an AI gateway the same thing?

An LLM gateway is a proxy layer between your applications and model providers. It exposes one API across many models and centralizes routing, failover, caching, rate limiting, and usage tracking, so reliability and spend are controlled in one place instead of per integration. You’ll also hear LLM proxy and AI gateway used for the same category.

The labels differ mostly by ambition. An AI gateway is the broader term vendors now prefer, since the same layer increasingly fronts embeddings, vision, and agent tool traffic, not just chat completions. In practice, the products in this guide answer to both names, and this page treats them as one category.

The distinction that does matter: an API gateway manages generic HTTP traffic and doesn’t natively understand tokens, models, or prompts. An LLM gateway adds token-aware rate limiting, per-model cost tracking, prompt caching, and multi-provider model routing. That’s why Kong, Cloudflare, and Apache APISIX all built AI-specific extensions rather than pointing customers at their existing proxies.

Why is an LLM gateway a finance decision, not just a platform one?

Here’s the CloudZero read on this category: the gateway is quietly becoming the most important line item control in the AI stack, and most companies are choosing one for reliability reasons without noticing they’re also choosing their metering point.

Start with what pushes teams here. It’s almost always one of three incidents, and often all three in the same quarter. The surprise invoice: the provider bill lands weeks after the usage, one lump sum, no way to say which team, feature, or customer drove it, because products often share a single API key.

The outage at the worst moment: a provider throws 5xx errors and every app wired directly to it goes down too, which is why multi-provider failover sits among the most cited adoption reasons in vendor and community writeups.

And the arbitrage left on the table: equivalent-quality tokens vary widely in price across providers, and hard-wired integrations can’t route the easy requests to a model that costs a tenth as much.

Agents turn all three from annoyances into budget events. Agentic workloads multiply calls per task, and Gartner projects over 40% of agentic AI projects will be canceled by the end of 2027, with escalating costs a leading cause. McKinsey’s 2026 survey adds the sharper number: about one in five organizations already say AI operating costs constrain their AI use. Cost is becoming the adoption ceiling before capability is.

Now for what makes this a finance decision and not just a platform one. Every request that crosses a gateway is visible at the moment it happens: the model, the tokens, the requester, and the cost, all in one place. That makes the gateway the cheapest place in your entire stack to measure AI spend, the raw input for AI cost per feature and per customer, and ultimately the evidence base for the AI ROI question your board is already asking.

Companies that treat the gateway as reliability plumbing get failover. Companies that treat it as their AI spend meter get failover plus the beginning of an answer to “what did we get for the money.”

How do you evaluate an LLM gateway on cost visibility?

Alongside the standard capabilities, every gateway here gets scored on four questions:

Scores reflect vendor documentation reviewed in September 2026, and this page is re-checked monthly, because this market refuses to sit still: in the last year alone TensorZero shut down, Portkey, Helicone, and archgw were all acquired, and Stripe agreed to buy OpenRouter for a reported $7 billion plus.

The best LLM gateways at a glance

Before the full reviews, the shortlist most teams end up choosing from. If you only read one table, the best LLM gateway candidates by deployment style are these:

The 21 gateways reviewed in full below get the same test. Honorable mentions follow with category and status only.

Best open source LLM gateways

Self-hosting an open source LLM gateway means it inherits your infrastructure, your compliance posture, and your existing observability. It also means the DevOps time is part of the price, whatever the license says.

1. LiteLLM

LiteLLM is the de facto standard: a Python proxy exposing an OpenAI-compatible API across 100+ providers, with virtual keys, per-key spend tracking, budgets, and tagging built in, which is why it doubles as a spend meter for many teams. The honest tradeoff is performance; Python shows its ceiling at high request volumes.

Cost-visibility test: strong. Best for teams that want control without a license fee and can budget the ops time.

2. Bifrost

Bifrost, the open source Go gateway from Maxim AI, is built for the traffic LiteLLM strains under: Maxim reports 11 microseconds of added overhead at 5,000 requests per second. It covers 20+ providers, ships an MCP gateway for agent tool traffic, and deploys in your VPC.

Newer ecosystem, smaller community, serious engineering.

Cost-visibility test: strong, with a per-request usage stream designed for downstream consumption. Best for production scale where gateway latency shows up in your p99.

3. Apache APISIX AI Gateway

Apache APISIX extends the veteran open source API gateway with AI proxy and governance plugins: provider routing, token-aware policies, and rate limiting alongside everything APISIX already does for regular APIs. If your platform team runs APISIX today, AI traffic becomes one more plugin rather than one more system.

Cost-visibility test: moderate; telemetry is solid, attribution takes assembly. Best for platform teams standardizing on one gateway for all traffic.

4. Higress

Higress is the AI-native gateway from Alibaba’s open source stable, an ingress controller grown into a full gateway with model routing and token-level controls. It carries real production lineage at scale and strong momentum in the Asia Pacific ecosystem, with thinner documentation outside Chinese-language channels.

Cost-visibility test: moderate. Best for Kubernetes-first teams comfortable in the cloud-native ecosystem.

5. Helicone

Helicone (YC W23) pairs its open source observability platform with a separate lightweight Rust gateway: drop-in proxy logging, caching, and routing with a one-line code change. The caveat now leads: Mintlify acquired Helicone in March 2026 and the standalone product runs in maintenance mode, with security updates, bug fixes, and new-model support continuing but feature development ended and Mintlify supporting migrations. The hosted tiers (free to 10,000 requests a month, then $79 Pro and $799 Team) remain live.

Cost-visibility test: strong on telemetry, lighter on enforcement. Best for teams already running it; new deployments should weigh a frozen roadmap against the durability advice at the end of this guide.

6. LLM Gateway

LLM Gateway does what the name promises: route, manage, and analyze LLM traffic through one API, self-hosted from the open source repo or via its hosted version. It’s earned adoption fast, helped and occasionally confused by carrying the category’s own name.

Cost-visibility test: moderate. Best for teams that want a lightweight router they can read end to end.

Best managed AI gateways

When self-hosting loses its charm, this is the tier: governance features that would take quarters to build, plus someone else’s pager.

7. Portkey

Portkey packs the deepest LLMOps feature set in the managed tier: guardrails, PII redaction, semantic caching, prompt management, audit logs, and observability across 1,600+ models, serving 24,000+ organizations. Pricing starts with a free Dev tier (10,000 logged requests a month) and scales by usage; note that past the log limit the gateway keeps routing but stops recording, so size the tier to your real traffic. Acquired by Palo Alto Networks in 2026.

Cost-visibility test: strong. Best for governance and cost controls out of the box.

8. Kong AI Gateway

Kong AI Gateway extends the API gateway your platform team may already run: AI plugins for routing, prompt security, and token-aware rate limiting, available open source and through Konnect managed plans. The pitch is consolidation; AI traffic gets the same discipline as the rest of your APIs. Starting fresh with no Kong footprint, other options fit faster.

Cost-visibility test: moderate. Best for existing Kong shops.

9. TrueFoundry

TrueFoundry aims at the enterprise end: a low-latency gateway the vendor rates at 350 requests per second on a single vCPU, fine-grained access control, in-VPC deployment, and governance features that map to procurement checklists, priced for enterprise procurement rather than five-person teams. It acquired Seldon AI in 2026 to extend its control plane.

Cost-visibility test: moderate-strong. Best for enterprises with a compliance team in the room.

10. Requesty

Requesty differentiates on operations: latency-based routing across 400+ models, automatic failover, prompt caching the vendor credits with 40 to 60% savings, five-layer RBAC, and the rare offer of dedicated EU hosting in Frankfurt with zero data retention. Vendor-reported numbers deserve your own benchmark, as always.

Cost-visibility test: strong. Best for teams with EU residency requirements that want routing managed for them.

11. Eden AI

Eden AI aggregates beyond language models: one API across LLMs plus vision, speech, translation, and other AI services, with unified billing. That breadth is the point and the limitation; it’s a general AI aggregator more than a deep LLM control plane.

Cost-visibility test: moderate. Best for product teams stitching several AI capabilities together who’d rather manage one bill than nine SDKs.

Best LLM routers and model marketplaces

An LLM router answers a different question than a proxy: not “how do I control my providers” but “which model should even get this request.” The best ones decide per prompt.

12. OpenRouter

OpenRouter is the fastest path from zero to every model: one key, 400+ models from more than 80 providers, unified billing, automatic fallback, and a free tier with daily caps. The convenience is priced as a credit fee around 5.5%, with no self-hosting and limited governance.

In August 2026, Stripe agreed to acquire OpenRouter for a reported $7 billion plus, a bet that token routing is becoming payments infrastructure; expect deeper billing integration and a new roadmap owner. Unbeatable for prototyping and model shopping; at enterprise volume, that fee line gets a hard look.

Cost-visibility test: moderate. Best for instant multi-model access.

13. Unify

Unify routes on benchmarks: it continuously measures quality, cost, and latency across providers and sends each request where your stated tradeoff points. It’s the most explicit about the quality-per-dollar frontier, which makes it a genuinely finance-legible router.

Cost-visibility test: moderate. Best for teams that want routing decisions justified by data rather than vibes.

14. Martian

Martian built its gateway around dynamic per-request routing to the best or cheapest capable model, with 200+ models behind one key and integrations reaching into the coding-tool wave: Cursor, Claude Code, Aider, and friends. Routing coding assistants through a cost-aware gateway is quietly clever: coding and developer tools captured $4.0 billion of enterprise AI spend in 2025, 55% of all departmental AI and the largest single category in the application layer, per Menlo Ventures.

Cost-visibility test: moderate. Best for cost-optimized routing, especially developer tooling.

15. Not Diamond

Not Diamond trains a router that picks the best-performing model for each individual prompt, learning from evaluation data rather than static rules. It’s the purest expression of the routing thesis: model choice as a per-request optimization problem, which also means you’re trusting its judgment with your quality bar, so evaluate on your own workload.

Cost-visibility test: basic. Best for optimizing quality-per-dollar at the prompt level.

Cloud platform AI gateways

If your stack already lives on one of these platforms, their gateway arrives with zero new vendors and one line of config. The tradeoff is ecosystem gravity.

16. Cloudflare AI Gateway

Cloudflare AI Gateway puts a proxy at the edge: caching, rate limiting, analytics, and logs across major providers, with core features free on a Cloudflare account and no token markup. Governance depth is modest and the value compounds mainly inside Cloudflare’s ecosystem.

Cost-visibility test: moderate; good analytics, lighter attribution. Best for teams already on Cloudflare that want caching and visibility for free.

17. Vercel AI Gateway

Vercel AI Gateway passes provider token prices through with no markup and layers on the operational kit: hundreds of models, fallbacks, BYOK, budgets that stack at team, project, and API-key scope, and a spend-reporting API that groups usage by model, user, tag, or key. Team-wide zero-data-retention enforcement costs $0.10 per 1,000 requests; per-request ZDR is free.

Cost-visibility test: strong; the budget and reporting primitives are genuinely finance-shaped. Best for Vercel and Next.js stacks.

18. Azure API Management AI gateway

Azure API Management’s generative AI gateway capabilities bring token-aware policies to the enterprise Azure estate: token limits, token-based throttling, semantic caching, and load balancing across Azure OpenAI deployments. It’s governance in the idiom Azure enterprises already speak.

Cost-visibility test: moderate, strengthening with Azure’s native cost tooling. Best for organizations whose AI runs through Azure architecture review.

19. AWS Bedrock intelligent prompt routing

Amazon Bedrock’s intelligent prompt routing keeps requests inside one model family but routes each prompt to the cheapest capable variant, with AWS citing savings up to about 30% without quality loss. Not a full multi-provider gateway, but for Bedrock-committed shops it’s cost routing with zero new infrastructure.

Cost-visibility test: moderate via AWS cost tooling. Best for AWS-native teams optimizing within Bedrock.

Framework-native gateways

The observability and eval platforms want the gateway seat too, bundling routing with the measurement layer you were adding anyway.

20. LangSmith Gateway

LangSmith Gateway builds runtime governance into LangChain’s platform: routing and controls living next to the traces and evals LangChain teams already use. If your stack is LangChain-native, gateway and observability collapse into one pane.

Cost-visibility test: moderate. Best for LangChain-first engineering organizations.

21. Braintrust proxy

Braintrust’s AI proxy takes the eval-platform angle: route through the proxy, and every request becomes loggable, cacheable, and evaluable against your quality baselines. A gateway for teams who believe model choice should be a measured decision.

Cost-visibility test: moderate. Best for eval-driven development shops.

Honorable mentions

Credible tools that didn’t need full reviews to earn a shortlist spot:

22. Gloo AI Gateway (Solo.io’s Kubernetes-native gateway for enterprise mesh environments)

23. Arize (observability platform with gateway capabilities)

24. FloTorch Gateway (governance-focused entrant)

25. archgw (the open source Arch gateway from Katanemo, acquired by DigitalOcean in 2026, where the project continues under the Plano name)

26. new-api (the most active gateway in the China ecosystem), and one-api, its long-running predecessor still in wide use.

Other platform-native entries include:

27. MLflow AI Gateway (routing inside the MLflow stack)

28. Databricks Mosaic AI Gateway (governance for Databricks-hosted models)

29. Tyk AI Studio (AI controls on the Tyk API platform)

30. Traefik Hub AI Gateway (AI routing in the Traefik ecosystem).

One you won’t find here: TensorZero, a well-regarded open source gateway, shut down in mid-2026. This category is consolidating fast. Factor vendor durability into your shortlist alongside features.

The gateway is the meter. You still need the ledger.

Run the cost-visibility test across all the tools above and a pattern emerges. The best gateways show you spend as it streams past: request by request, token by token, priced to the fraction of a cent.

What none of them does is turn that stream into what finance actually needs weeks later, when the provider invoice lands as a lump sum: spend allocated to products, features, teams, and customers, reconciled against the bill, and expressed as unit economics a board can act on. A meter reading isn’t a P&L answer. It’s the input to one.

That’s the gap CloudZero closes, and it’s why the gateway you just shortlisted is also your fastest path to proving AI ROI.

CloudZero’s AI Signals ingests usage telemetry straight from the gateway layer: there’s a LiteLLM integration and a Bifrost collector adapter, plus OpenTelemetry for everything else, with live spend visible within minutes instead of when the invoice arrives.

From there, gateway traffic joins the rest of your cloud and AI spend in one place: allocated to the products and customers that drove it, budgeted, tuned for efficiency, and reconciled from token to invoice.

Alongside the gateway collectors, CloudZero’s connectors cover Anthropic, OpenAI, and the rest of the stack around them, and the AI cost management landscape if you’re mapping the full toolchain.

So the playbook: pick the gateway that fits your stack from the list above, turn on its telemetry, and wire the stream into a system that makes the spend answerable. Meter first, ledger second, surprise invoices never.

This isn’t hypothetical. One employee at Hudl was driving $600K per year in token spend across 40 different models, 70% of the company’s total AI costs, and nobody in finance or engineering knew, because the spend surfaced as shared compute with no attribution. A gateway would have caught the pattern. A cost platform tells you it’s one person, which team, and whether it was worth it.

Ready to wire the meter to the ledger? Schedule a demo to see gateway telemetry flowing into cost per feature and per customer. Prefer to poke around first? Take the self-guided product tour. Or start with a free cloud cost assessment and find out what your AI spend is hiding today.

Frequently asked questions about LLM gateways

Author Spotlight

Lyne Carolyne

Lyne Carolyne has several years of experience in AI and cloud economics and brings that understanding into the content she creates. Outside work, she's an avid explorer.