Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism

Engineers from NVIDIA have detailed an experimental framework, Nonuniform Tensor Parallelism (NTP), designed to address transient hardware failures during large-scale AI model training. The technique allows training jobs to dynamically adapt to GPU unavailability without significant performance degradation. This is critical as training runs extend across thousands of GPUs, where even minor interruptions can stall the entire system, impacting the overall efficiency, or Goodput, which measures useful computational work rather than raw throughput.

Maintaining Performance Under Duress

NTP operates by reconfiguring a training workload in real-time when a GPU within a tensor parallelism (TP) group becomes unavailable. Instead of halting or dropping the entire data parallel replica, NTP enables the group to continue processing with the remaining healthy GPUs. To prevent this smaller group from becoming a system-wide bottleneck, the framework proposes a novel hardware co-design feature: dynamic power boosting. This allows the remaining active GPUs to temporarily operate at higher clock frequencies to compensate for the reduced resources and maintain pace with other replicas. The necessary redistribution of model data, or resharding, is efficiently overlapped with other computational steps, introducing an overhead of less than 1%.

The core mechanisms of NTP include:

• Dynamic TP Degree Adaptation: If a GPU in an 8-GPU TP group fails, the group can instantly switch to a 7-GPU configuration and continue its work.
• Performance Compensation via Power Boosting: The remaining GPUs in the affected group receive increased power, boosting their clock speeds to prevent the replica from slowing down the entire data-parallel job.
• Efficient Overlapped Resharding: Tensor redistribution is performed during the backward computation and parameter sync phases to hide the latency cost.

The approach underscores the growing importance of hardware and software co-design for building resilient AI infrastructure, particularly for next-generation systems like NVIDIA's Blackwell and Blackwell Ultra platforms, which feature scale-up domains of up to 72 GPUs connected via NVIDIA NVLink. While still experimental, NTP has been added to the developer branch of NVIDIA Megatron Core, signaling a move toward more inherently fault-tolerant training systems. Research is also extending these principles to Mixture-of-Experts (MoE) models with a concept called Nonuniform Expert Parallelism (NEP).

NTP signals a shift from reactive fault tolerance, such as checkpoint-restarts, to proactive in-flight resilience, treating transient GPU failures as manageable performance variations rather than catastrophic job-stalling events. This co-design of hardware power headroom and adaptive software is critical for optimizing the TCO of next-generation AI factories.