AWS adds GPU diagnostics to Amazon SageMaker HyperPod
AWS introduced enhanced observability for Amazon SageMaker HyperPod, aimed at teams tracking down GPU problems in large distributed training jobs. The capability runs on clusters orchestrated by Amazon EKS or Slurm and is installed as an add-on or extension. It gathers default metrics every 30 seconds, with optional advanced metrics covering NVIDIA DCGM, AWS Neuron, host-level node exporter and Elastic Fabric Adapter network telemetry, sending them to CloudWatch, now the default store, or Amazon Managed Service for Prometheus. A diagnostic summary uses CloudWatch's DetectGroupOutliers API to flag GPUs that deviate from peers on utilization, memory, temperature, power and clock speed, while static rules catch XID errors, ECC memory faults, EFA throughput drops and thermal throttling, identifying the affected job, node and rank.
Why it matters
When throughput drops on a multi-GPU job, teams can now see which specific accelerator or network link is failing instead of stitching together separate dashboards and logs, which AWS says shortens troubleshooting and keeps expensive hardware in use.
Sources
- PrimaryAWS Machine Learning
Written by AIToolMaps from the source above and checked against it sentence by sentence.