Google puts AI model, token and tool rules into Envoy
Envoy limits traffic by tokens and not by requests, and it sends each agent turn to the server that holds its cache.
Google Cloud engineers tell how Envoy, an open source proxy that sends traffic between apps and servers, works with AI traffic. Google names Envoy as a possible open standard for AI traffic, tool calls and agent to agent communication. It tells that tokens, the units of text that a model reads, set the cost of traffic.
What is new
Envoy has these functions:
- Token limits for each tenant, project or model.
- A turn goes to the GPU worker that holds its cache.
- A retry on a different model when the first model gives an error.
- Checks of tool calls and prompts by tools such as Model Armor.
The numbers
Google gives no test results and no speed figures. It gives one example: a prompt of 500,000 tokens costs far more than a short query. It tells that a change from a model of a vendor to a model on your servers is only a configuration change.
What is not known
All claims are from Google. For security checks it tells only of low latency and gives no number. It does not tell how many teams use these functions.
Insight: Google wants the traffic gate for agents to be open source
In October 2026, three AI companies each gave an agent tool to one neutral group. Google puts the rules for agent traffic into Envoy, which is open source. This reads as a race to control the gate that all agent calls use, and leaders must find who controls it.
Sources
Posted