Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine

NVIDIA Optimizes JAX for 10x Faster MoE Model Training

NVIDIA has released a series of optimizations for the JAX framework that deliver a 10.4x throughput improvement for training Mixture of Experts (MoE) models. By integrating its Transformer Engine library, the company demonstrated that training performance for the DeepSeek-V3 671B model on NVIDIA GB200 hardware increased from a baseline of 103 to 1,068 TFLOPS per GPU. This addresses a critical efficiency bottleneck for 'dropless' MoE architectures, which preserve model quality by processing every token but have historically presented significant performance challenges due to irregular workloads.

The performance gains stem from a full-stack approach targeting the unique problems of MoE training, such as uneven token distribution to experts, which creates inefficient 'ragged tensors'. Instead of forcing models to drop or pad tokens to fit uniform hardware constraints, NVIDIA's new software path adapts to the dynamic nature of the workload. Key enhancements are now available through the NVIDIA NGC MaxText container and include:

• Grouped GEMM Kernels: A single kernel call within Transformer Engine that processes matrix multiplications for all experts at once, each with its actual, variable token count, eliminating padding-related waste and slow kernel loops.
• Optimized Expert Parallelism: Powered by NCCL EP, this fuses the data dispatch and combine stages and uses token deduplication to significantly reduce network traffic during all-to-all communication.
• JAX and XLA Enhancements: System-level improvements include JAX host offloading to conserve high-bandwidth memory and XLA multistreaming collectives, which overlap communication across NVLink and InfiniBand fabrics to hide latency.

These optimizations enable training to scale with high efficiency, which is crucial for the massive computational budgets required for next-generation foundation models. NVIDIA reports sustaining 97% scaling efficiency at 1,024 GPUs on a GB300 NVL72 system. By solving the core performance issues of dropless MoE, this work makes it more practical for developers in the JAX ecosystem to train higher-quality sparse models without compromising on architectural integrity or computational efficiency, solidifying the company's integrated hardware and software advantage in the AI infrastructure market.

The focus on 'dropless' MoE is a strategic move to couple hardware performance directly with model quality. Instead of forcing architects to compromise on training data by dropping tokens for hardware convenience, NVIDIA is adapting its software stack to the ideal model architecture. This tight integration of hardware-aware kernels and framework-level optimizations creates a deep moat, making it difficult for competitors to match performance without a similarly vertically integrated approach.