Skip to content

AWS adds one endpoint routing across HyperPod GPU clusters

Short answer

AWS introduced the Global Inference Gateway for SageMaker HyperPod. It gives teams one managed endpoint in front of all their HyperPod clusters, in one Region or several. It scores clusters on load and errors, sends each request to the best one, and fails over automatically. It is available in 13 AWS Regions.

What this means for operators

This matters only if your company already hosts its own models on HyperPod clusters, for example an internal assistant or a support-triage model running on rented GPUs. If so, the gateway removes the per-cluster endpoint list your integrations carry and the manual traffic switch when a cluster goes down. Your CRM, helpdesk and workflow automations can call one signed endpoint. If you use hosted model APIs, nothing changes for you. Before adopting it, check that every cluster serving a model is configured the same way, including identity-token checks, and note that published clusters not attached to any gateway still incur load balancer charges.

AWS has introduced the Global Inference Gateway for Amazon SageMaker HyperPod, a managed gateway that sits above all the HyperPod clusters a team registers. It is the second tier of an existing design. The per-cluster Inference Gateway still chooses the pod inside each cluster; the new layer chooses the cluster.

The problem it targets

AWS describes a common situation: one cluster is saturated and queuing requests while GPUs on another cluster, sometimes in another Region, sit idle. Its example is a team serving four models on five clusters in two Regions. Before the per-cluster gateway, clients tracked 20 endpoints. The per-cluster gateway cut that to five, but each client still had to pick a cluster, and failover meant an engineer editing configuration by hand.

How it works

Setup has two steps. First, each cluster is published by enabling a global routing block in its existing gateway configuration; the controller then creates a Network Load Balancer and a VPC endpoint service in the customer's own account. Second, the team calls CreateGlobalInferenceGateway with the list of clusters and receives a single endpoint.

For every request the gateway scores each healthy cluster on four signals: available KV cache and GPU memory, queue depth, distance from the gateway's Region, and recent error rate. The defaults weight these at 0.20, 0.25, 0.40 and 0.15. Teams can change the weights, which must sum to 1.0, and AWS suggests raising the error weight if resilience matters more than proximity.

Failover and security

When a cluster fails, traffic moves to the next-best cluster and returns gradually after recovery. A failed request is retried once on another cluster unless streaming has started. If no healthy cluster remains, the gateway returns a 503. Clients send OpenAI-compatible requests signed with SigV4 and authorized through IAM, and traffic reaches clusters only over AWS PrivateLink. AWS states the gateway does not log or store prompts or completions.

Caveats from the post

  • A published cluster that belongs to no gateway gets no traffic, but its load balancer keeps running and is billed.
  • A model missing from every scheduler in the gateway configuration cannot be reached and returns a 404.
  • Identity-token checks should be on for all clusters serving a model or for none, so a request does not pass on one cluster and fail on another.
  • A gateway can be limited to one Region by including only clusters there.

The source does not give a separate price for the gateway. The service is available in 13 AWS Regions across the Americas, Europe and Asia Pacific.

Source: AWS Machine Learning Blog

Next step

Visibility Analyzer

An AI news item will not tell you how assistants see your own site. The Visibility Analyzer checks that on your live site: which answers cite you, which pages an engine cannot retrieve, and what to fix first. Free to run.

Run a free visibility audit

Free to run. No card.

Fee
Free
Length
One run, minutes

Free tier: two analyses a day, no card required.