AWS adds Global Inference Gateway to Amazon SageMaker HyperPod
AWS introduced the Global Inference Gateway for Amazon SageMaker HyperPod, a SageMaker-managed endpoint that routes inference requests across every HyperPod cluster a customer registers, whether those clusters sit in one AWS Region or several. Customers publish each cluster and then call CreateGlobalInferenceGateway with an execution role; the service builds the cross-Region connectivity and carries requests over AWS PrivateLink, authenticating callers with SigV4 and IAM.
The gateway health-checks clusters, shifts traffic off a failed one automatically, and scores healthy clusters using adjustable weights on KV cache headroom, queue depth, Region proximity and error rate. Each cluster's own Inference Gateway still selects the pod, and clients send OpenAI-compatible requests to the single endpoint. Weights can be changed later with UpdateGlobalInferenceGateway.
Why it matters
Teams that spread GPU capacity across clusters and Regions can now use idle capacity during traffic peaks instead of building and maintaining their own load balancers, DNS and cross-Region networking, and failover no longer requires an engineer to reconfigure clients.
Sources
- PrimaryAWS Machine Learning
Written by AIToolMaps from the source above and checked against it sentence by sentence.