Google adds streaming to its API Gateway in a public preview
Streaming connections can stay open for a maximum of one hour behind the gateway and keep its security controls.
Google gave API Gateway, its Cloud service in front of the APIs (the connection points of software) of a company, streaming support in a public preview. Before this, a team had to give up streaming, or send streaming traffic around the gateway and give up its key checks and logs. Google tells that AI apps must have this, as they send model answers in pieces.
How it works
Streaming endpoints (service addresses) operate behind the gateway. The API specifications do not change, and the gateway keeps its policies, such as API key checks and logs. One endpoint can also send work to 3 model families, selected by the model field in the query:
The numbers
A streaming connection can stay open for a maximum of one hour. After that, the client must open a new connection. Google gave no speed data or test results.
What is not known
In the preview, a developer must start streaming with the flag enable-streaming (a part of the command). Users on social media point out 2 limits:
- A gateway that is in use cannot change to streaming.
- Latency charts show the time of the full stream, not the time to the first byte.
Insight: The gateway becomes the control point for AI traffic
Google adds streaming and model routing to one gateway. A Cloudflare beta put web search, logs and the bill for agents through its gateway. This reads as a race for the gateway, because the owner can see each query and set which models a team can select. The gateway that you select can limit the models that you select.
Sources
Posted
