NVIDIA releases open-source Recovery service for distributed training

NVIDIA published NVIDIA Recovery, an open-source, workload-level service that watches a cluster's training jobs and automates a four-stage recovery loop: detecting crashes, hangs and stragglers from application logs, GPU metrics and scheduler signals; attributing the failure; isolating the implicated nodes; and then restarting or stopping workloads according to operator policy. It sits in the company's DSX OS software stack and includes a Slurm integration through a reference Advisor Controller installed via Helm chart in either in-cluster or SSH mode. Source code and an issue tracker are in the dsx-ai-factory/NVIDIA-Recovery repository, alongside a quickstart that runs locally without a physical GPU and an operator deployment guide. New deployments begin in monitor mode, publishing recommendations without taking scheduler actions.