Best LLM Gateways and Agent Platforms With Built-In Gateways in 2026

Quick answer

Choose a standalone LLM gateway if you only need to control model traffic. Choose an agent platform with built-in routing or a gateway if you also want agents, evals, and monitoring in one place.

• Open source self-hosting: LiteLLM
• Hosted access to many models: OpenRouter
• Open source plus a managed product: Portkey
• Enterprises already using Kong: Kong AI Gateway
• Teams already on Cloudflare: Cloudflare AI Gateway
• LangSmith users: LangSmith LLM Gateway, currently in beta
• AWS agent tool access: Amazon Bedrock AgentCore Gateway, which is a different kind of gateway
• Managed agents with per-task model routing: Brainbase Router

Disclosure: Brainbase publishes this guide and is one of the options compared. We reviewed public documentation and pricing on November 5, 2026. We did not run a load test or latency benchmark.

Standalone gateway or built-in routing?

A standalone gateway sits between your applications and model providers. It works for every app, but it doesn't know what your agent was trying to accomplish.

Built-in routing sits inside the agent platform. It can use information about the agent and task, but it may only serve agents on that platform.

Use a standalone gateway when many applications share model access, or when a central team needs one control point for keys and policies.

Use built-in routing when agents are the main workload and you want routing, traces, and evals in one system.

You can use both. A central gateway can enforce company policy, while the agent platform chooses a model for each task. Before you combine them, check that retries don't happen twice and that costs aren't counted twice.

One naming warning: not every "gateway" is mainly about model traffic. AgentCore Gateway mainly turns APIs and Lambda functions into agent tools, although it can also set usage limits. Check what the product actually controls.

How the options compare

Hosting and cost controls are the main differences. Each entry reflects public vendor pages on November 5, 2026. "Not stated" means we didn't find it on the pages reviewed, not that the feature is missing.

Which option fits which team

LiteLLM

LiteLLM is a self-hosted, OpenAI-compatible gateway for more than 100 providers. It supports load balancing, fallbacks, retries, and virtual keys with budgets. It suits teams that want control and can operate the service and its Postgres database.

Portkey

Portkey offers open source self-hosting and paid plans, and its pricing page now presents it as Prisma AIRS AI Gateway. Routing and fallbacks appear on all plans, while VPC deployment and granular budgets are on Enterprise.

OpenRouter

OpenRouter is a hosted service with a platform fee on paid plans. It suits teams that want quick access to many models. Compare the fee with a self-hosted option as your volume grows.

Kong AI Gateway

Kong AI Gateway manages LLM, MCP, and A2A traffic, with failover, token quotas, cost budgets, and semantic caching. Data planes can be managed by Konnect or self-hosted. It is a natural fit for enterprises that already run Kong.

Cloudflare AI Gateway

Cloudflare AI Gateway adds caching, rate limits, retries, and model fallback to model traffic. Dynamic routes and spend limits are in beta. It suits teams already building on Cloudflare.

LangSmith LLM Gateway

LangSmith LLM Gateway is a beta feature that accepts OpenAI and Anthropic request formats. It offers fallbacks, spend and rate limits, and data policies. It suits teams that already use LangSmith.

Brainbase Router

Brainbase is an agent platform rather than a standalone gateway. Its Router picks a model per task. It suits teams that want routing tied to managed agents, evals, and traces.

It isn't a replacement for a central gateway serving all of your company's applications.

How to test routing, fallback, and spend limits

Test each gateway by forcing the failures you expect in production. A feature list won't show what happens when a provider goes down at peak traffic.

1. Routing: Send the same mix of real tasks through each option. Record the model chosen, the latency, the cost, and the quality of the result.
2. Fallback: Use a bad key, a blocked provider, and a forced timeout. Check that the request fails over, and that the fallback model can do the job.
3. Streaming: Fail a provider mid-stream. Check whether the user sees a duplicate, a partial reply, or an error.
4. Tool calls: Run an agent through fallback. Tool-call formats can vary by model, so confirm the agent still completes the task.
5. Spend limits: Set a low budget, then exceed it. Check whether the gateway blocks the request or only reports it later.
6. Rate limits: Exceed team and key limits, then confirm that the error is clear and that one team can't starve another.
7. Audit: Trace one request from the application to the provider. Confirm who made it, what it cost, and where the data went.

Also test a gateway outage. If every agent depends on it, the gateway becomes a single point of failure.

FAQ

What is an LLM gateway?

An LLM gateway is a layer between your applications and model providers. It typically handles routing, fallback, keys, limits, and logging.

Do I need a gateway if I use one model?

Probably not at first. A gateway becomes more useful when you add providers, teams, budgets, or compliance rules.

Can a gateway run on-premise?

Yes, several can. LiteLLM is self-hosted, and Kong supports self-hosted data planes. Check whether the model providers you use also meet your data rules.

Is an agent gateway the same as an LLM gateway?

Not always. An LLM gateway controls model traffic. An agent gateway may control tool, MCP, or agent-to-agent traffic.

Does a gateway reduce AI costs?

It can, through routing, caching, and limits. It also adds a service or fee, so measure the total cost.

Sources

Reviewed November 5, 2026. Check current vendor documentation before you decide.

• LiteLLM proxy docs
• Portkey pricing
• OpenRouter pricing and provider routing docs
• Kong AI Gateway docs
• Cloudflare AI Gateway docs
• LangSmith LLM Gateway docs
• Amazon Bedrock AgentCore FAQs
• Brainbase home and pricing