AWS has introduced the Global Inference Gateway for Amazon SageMaker HyperPod, a managed gateway that sits above all the HyperPod clusters a team registers. It is the second tier of an existing design. The per-cluster Inference Gateway still chooses the pod inside each cluster; the new layer chooses the cluster.
The problem it targets
AWS describes a common situation: one cluster is saturated and queuing requests while GPUs on another cluster, sometimes in another Region, sit idle. Its example is a team serving four models on five clusters in two Regions. Before the per-cluster gateway, clients tracked 20 endpoints. The per-cluster gateway cut that to five, but each client still had to pick a cluster, and failover meant an engineer editing configuration by hand.
How it works
Setup has two steps. First, each cluster is published by enabling a global routing block in its existing gateway configuration; the controller then creates a Network Load Balancer and a VPC endpoint service in the customer's own account. Second, the team calls CreateGlobalInferenceGateway with the list of clusters and receives a single endpoint.
For every request the gateway scores each healthy cluster on four signals: available KV cache and GPU memory, queue depth, distance from the gateway's Region, and recent error rate. The defaults weight these at 0.20, 0.25, 0.40 and 0.15. Teams can change the weights, which must sum to 1.0, and AWS suggests raising the error weight if resilience matters more than proximity.
Failover and security
When a cluster fails, traffic moves to the next-best cluster and returns gradually after recovery. A failed request is retried once on another cluster unless streaming has started. If no healthy cluster remains, the gateway returns a 503. Clients send OpenAI-compatible requests signed with SigV4 and authorized through IAM, and traffic reaches clusters only over AWS PrivateLink. AWS states the gateway does not log or store prompts or completions.
Caveats from the post
- A published cluster that belongs to no gateway gets no traffic, but its load balancer keeps running and is billed.
- A model missing from every scheduler in the gateway configuration cannot be reached and returns a 404.
- Identity-token checks should be on for all clusters serving a model or for none, so a request does not pass on one cluster and fail on another.
- A gateway can be limited to one Region by including only clusters there.
The source does not give a separate price for the gateway. The service is available in 13 AWS Regions across the Americas, Europe and Asia Pacific.