What matters in AI.

Subscribe

Google puts AI model, token and tool rules into Envoy

Envoy limits traffic by tokens and not by requests, and it sends each agent turn to the server that holds its cache.

Google Cloud engineers tell how Envoy, an open source proxy that sends traffic between apps and servers, works with AI traffic. Google names Envoy as a possible open standard for AI traffic, tool calls and agent to agent communication. It tells that tokens, the units of text that a model reads, set the cost of traffic.

What is new

Envoy has these functions:

  • Token limits for each tenant, project or model.
  • A turn goes to the GPU worker that holds its cache.
  • A retry on a different model when the first model gives an error.
  • Checks of tool calls and prompts by tools such as Model Armor.

The numbers

Google gives no test results and no speed figures. It gives one example: a prompt of 500,000 tokens costs far more than a short query. It tells that a change from a model of a vendor to a model on your servers is only a configuration change.

What is not known

All claims are from Google. For security checks it tells only of low latency and gives no number. It does not tell how many teams use these functions.

Insight: Google wants the traffic gate for agents to be open source

In October 2026, three AI companies each gave an agent tool to one neutral group. Google puts the rules for agent traffic into Envoy, which is open source. This reads as a race to control the gate that all agent calls use, and leaders must find who controls it.

Sources

  1. Building an agentic stack in Envoy: routing models, tokens, and MCPcloud.google.com
AI MATTER · NEWS · AI MATTER · NEWS ·18 NOV2026

Posted

Tags

Learn the terms in this story

More in Agents

All Agents news