ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding

NVIDIA and Princeton Detail ReplaySSM to Boost Hybrid AI Model Speed

Researchers from NVIDIA and the Dao AI Lab at Princeton University have detailed ReplaySSM, a technique designed to address a critical performance bottleneck in hybrid large language models. By rethinking how state-space models (SSMs) manage their state during decoding, the method significantly reduces memory traffic and enhances the efficiency of speculative decoding. This is particularly relevant for the growing class of models that interleave SSM layers with attention layers, such as NVIDIA Nemotron 3 and Qwen3.5, which can become memory-bound at the large batch sizes typical of production environments.

Conventional SSM decoding loads and stores the entire model state from high-bandwidth memory at every step, creating a latency issue. ReplaySSM decouples output computation from state updates by representing the state as a stable, high-precision checkpoint plus a small buffer of recent inputs. The main state checkpoint is only written back to memory when the buffer is full, an operation called a flush. This approach delivers substantial performance gains on modern hardware like the NVIDIA Blackwell Ultra B300 GPUs.

• Reduced Memory Traffic: The technique cuts state memory traffic by approximately half.
• Faster Kernels: SSM decoding kernels are accelerated by 1.46x to 1.80x at a batch size of 256.
• Improved Speculative Throughput: Speculative decoding throughput increases by 1.85x to 2.10x on 120B-class models.
• Efficient Quantization: It supports low-precision state reads for output computation while using a high-precision checkpoint for updates, limiting the accumulation of quantization errors.

Ecosystem Adoption and Market Impact

The practical benefits of ReplaySSM are underscored by its swift integration into major inference frameworks, including NVIDIA TensorRT-LLM, vLLM, and FlashInfer. Furthermore, it is already being used in production models like GLM-5.3-Flash and Kimi-K3, signaling its immediate utility for deploying large-scale AI services. By making the SSM layers faster and more compatible with essential optimization techniques like speculative decoding and quantization, ReplaySSM strengthens the case for hybrid architectures as a competitive design for building efficient, long-context language models.

ReplaySSM is a critical optimization that bridges the gap between the theoretical efficiency of state-space models and their practical performance in production, solidifying hybrid architectures as a viable path for scalable, long-context AI.