What matters in AI.

Subscribe

Amazon adds one gateway that sends AI requests across GPU clusters

The gateway sends each request to the best cluster in each Region, and moves traffic off a cluster that stops.

Amazon Web Services (AWS) added the Global Inference Gateway to SageMaker HyperPod. This system for GPU clusters operates AI models. One gateway sends requests across all clusters that a team registers, in one Region or in many. AWS gives an example: four models on five clusters give 20 endpoints (network addresses) today, and one gateway replaces them.

How it works

A team publishes each cluster one time. Then it makes one gateway with the clusters. The gateway scores each healthy cluster on four signals and picks the best. It sends the request on the AWS private network, and it checks the caller with AWS sign-in and permission controls.

The team sets weights from 0.0 to 1.0 for the four signals. The weights must be 1.0 in all.

The numbers

AWS gave no speed or cost results. Its one customer proof is words from Bagelcode, a game company. Bagelcode says that in a beta test, requests went across two Regions. They continued when one cluster could not use the model.

What is not known

AWS did not give the price of the gateway. It did not say how much GPU capacity the gateway recovers. It gave no date for general release. A team must publish each cluster before it lists the cluster in a gateway.

Insight: AWS now routes by capacity at two layers

AWS sells Bedrock, a system with models, and in October 2026 it added cross-Region inference for the US government cloud. This sends each request to the Region that has capacity. At this time the same idea works for clusters that customers own. This reads as AWS keeping capacity decisions in its own routing layer, and customers use it more. Leaders must find how easily they could move that routing to a different vendor.

Sources

  1. Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clustersaws.amazon.com
AI MATTER · NEWS · AI MATTER · NEWS ·20 NOV2026

Posted

Tags

Learn the terms in this story

More in Compute

All Compute news