How NVIDIA Mission Control 2.5 Brings Autonomous Recovery to Kubernetes

Mission Control Now Manages Kubernetes Resiliency

NVIDIA has released Mission Control 2.5, extending its autonomous recovery engine to Kubernetes environments managed by NVIDIA Run:ai. This expansion, previously available only for Slurm-based HPC clusters, addresses a critical operational need for modern AI factories. As distributed training jobs run for days or weeks across hundreds of nodes, the ability to proactively manage hardware anomalies and software errors becomes essential to prevent costly interruptions and maximize GPU utilization.

A Three-Layered Recovery Framework

The system operates as an integrated three-layer control loop that separates workload placement from failure analysis and hardware remediation. This allows for more intelligent recovery actions, such as retrying a job after a transient software error without unnecessarily quarantining a healthy node. The framework is designed to reduce downtime by enabling parallel recovery tracks, where a workload can be rescheduled on healthy nodes while diagnostics run on the isolated faulty hardware. The core components include:

• Kubernetes Scheduler: NVIDIA Run:ai manages job placement, queuing, and scheduling policies onto available compute capacity.
• Autonomous Job Recovery: This service analyzes job context and failure history to classify the type of error and determine the correct recovery path, such as a restart, retry, or re-queue.
• Autonomous Hardware Recovery: This layer monitors infrastructure health, removing faulty or at-risk nodes from the scheduling pool and initiating diagnostics to prevent failures from impacting new jobs.

Unifying AI Operations

By offering a consistent recovery model for both Kubernetes and Slurm, NVIDIA is standardizing the operational backbone for organizations running mixed AI workloads. This allows platform teams to manage their entire accelerator estate with a unified workflow, moving from reactive maintenance to a proactive, self-healing model. For data center operators, this means faulty nodes are automatically isolated and diagnosed, while AI practitioners experience simpler, more automated job recovery without needing to intervene in underlying infrastructure issues.

By abstracting resiliency away from the specific workload orchestrator, NVIDIA is positioning Mission Control as an essential operational layer for any at-scale AI factory. This reinforces the value of its full-stack ecosystem and makes it more difficult for competitors to offer piecemeal infrastructure management solutions.