AI gateway

How it works

The app sends its model calls to the gateway, not to the provider. The gateway applies its rules. Then it calls the provider with a key that the app does not see.

Three AI gateways show six groups of control:

Control	Azure API Management	Cloudflare AI Gateway	LiteLLM
Keys	Managed identity, no API keys	Stored provider keys (BYOK)	Virtual key for each team
Limits	Token limit for each minute or period	Rate and spend limits	Budgets and rate limits
Cost	Token metrics for each consumer	Analytics, custom costs	Cost of each call
Speed	Semantic caching	Cache of the same calls	Not on the page we used
Resilience	Load balancer, circuit breaker	Retries, model fallback	Load balancing, fallbacks
Content	Azure AI Content Safety	Guardrails, data loss prevention	Guardrails, PII masking

Sources: vendor documentation, read on 13 November 2026.

Why it matters

An AI gateway gives a security team one point to see and control model use.

• Keys stay in one point. Envoy AI Gateway “keeps provider credentials out of your application”.
• Cost has a limit. Token limits and spend budgets stop a runaway task or a stolen key. This is a control against cost harvesting.
• Prompts are on record. Logs of prompts and responses give an audit trail.
• One policy for all models. Content checks such as AI guardrails operate at the gateway, for all providers.

The same position makes the gateway a target. It reads all prompts and holds all provider keys.

In practice

Some AI gateways also front more than models. Azure API Management, LiteLLM and Envoy AI Gateway put MCP servers behind the same endpoint. Envoy AI Gateway 1.0 shipped on 23 June 2026. Its name is Agent Router at this time.

Three risks come with an AI gateway. Each risk has a different control.

1. A third-party router that you do not control. Researchers tested 28 paid and 400 free LLM API routers in 2026. One paid and 8 free routers injected malicious code. 17 touched the researchers’ AWS canary credentials. This is a malicious LLM proxy router. Operate the gateway on your own systems, or use a provider with a contract.
2. A compromised gateway package. The LiteLLM attack was an AI supply chain compromise. Users of the LiteLLM Proxy Docker image were not affected, because its dependencies were pinned. Pin versions and check them.
3. A router that an attacker can steer. Shafran and colleagues (January 2025) made “confounder gadgets”. These token sequences send a query to the strong, expensive model. They worked on open-source and commercial routers. Perplexity filters did not stop them.

The routing benefit has limits. The RouteLLM paper reports more than 2 times lower cost “in certain cases”, on benchmarks. Its routers select between one stronger and one weaker model.

Questions and answers

What is the difference between an AI gateway and an API gateway?

An API gateway controls API calls in general. An AI gateway adds controls for model traffic. It counts tokens, not only calls. It can also check prompt content and change between model providers. Some AI gateways are part of an API gateway. Microsoft says that its AI gateway extends the API Management gateway and is "not a separate offering".

Can an AI gateway pick the best model for each task?

Some can. Cloudflare says its Auto Router picks the most cost-effective model for each request. The RouteLLM paper reports more than 2 times lower cost in some conditions, with no loss of quality. A 2025 study showed that added token sequences can trick routers into the expensive model.