ProductsNov 18, 2026, 6:03 PM EST

AWS adds workload recovery and new health checks to SageMaker HyperPod

Amazon SageMaker HyperPod now includes three capabilities aimed at training jobs that stall or fail even when node hardware looks healthy. AWS describes a MultiNodeTrainingValidation deep health check that runs a short Megatron-LM training job on each instance and then across instances, failing hardware whose loss turns NaN or infinite or whose run does not finish in time. A Kubernetes resource monitoring agent on EKS clusters reports crash-looping device plugins, missing DaemonSets, tainted nodes and the pods and nodes behind a failed job. Hung job detection for Slurm watches job output and GPU activity and can cancel a stalled job so Slurm requeues it. The deep health check runs during provisioning or on demand on supported EFA-enabled NVIDIA GPU instances.

Why it matters

Long training runs can burn GPU hours when a job silently stops progressing or a GPU returns bad results while passing hardware tests. These checks aim to flag those failures sooner and, with automatic node recovery or requeueing, let the job continue instead of sitting idle.

Sources

Written by AIToolMaps from the source above and checked against it sentence by sentence.

Comments

No comments yet