Accelerating Mamba-2 with High-Performance Forward Pass Kernels in NVIDIA CUDA Tile

Jane Street and NVIDIA Optimize Mamba-2 Using CUDA Tile

Quantitative trading firm Jane Street has partnered with NVIDIA to accelerate the Mamba-2 state-space model, utilizing the NVIDIA CUDA Tile Python-based DSL to craft high-performance forward pass kernels. This work addresses a core challenge in sequence modeling by combining the parallel training advantages of transformers with the inference efficiency of recurrent architectures. By implementing Mamba-2's complex recurrence in cuTile, the engineers demonstrate a viable path for developing novel model architectures that can effectively leverage modern GPU hardware like Tensor Cores without resorting to low-level CUDA C++.

The technical implementation centers on three custom kernels designed to handle the Mamba-2 forward pass in a chunkwise, parallel manner. This approach avoids materializing the massive hidden state tensor typical of parallel scan algorithms, which is often memory-bound. Instead of a simple for-loop, the team used a 'jump formula' to compute future hidden states with matrix multiplications, making the process friendly for Tensor Cores. A key insight involves reformulating the model's exponential decay term in log-space, allowing sequential multiplications to be replaced with more efficient cumulative sums, which are pre-calculated and stored by the first kernel.

This collaboration signals a significant development for the broader AI ecosystem, as tools like CUDA Tile make GPU performance engineering more accessible to machine learning specialists who primarily operate in Python. By abstracting away complex thread- and warp-level scheduling behind high-level, tile-centric primitives, such DSLs lower the barrier to creating custom, hardware-aware operators. This can accelerate the exploration and adoption of non-transformer architectures, fostering innovation in model design that is both computationally efficient and highly performant.

Key Kernel Optimizations

• chunk_cumsum: Computes per-token, per-head cumulative sums of the decay term in log-space within each data chunk, storing the results for reuse by subsequent kernels.
• fwd_pass_state: Executes Tensor Core matrix multiplications (ct.mma) to advance the hidden state across sequence chunks, applying the model's jump formula with pre-multiplied operands.
• compute_out: Fuses multiple operations—including the carried-state term, the decayed and causally masked interaction matrix, and the skip connection—into a single tiled kernel to minimize memory traffic and maximize register use.

Strategic Takeaway

The collaboration between Jane Street and NVIDIA underscores a critical industry trend: abstracting away the immense complexity of modern GPU hardware through high-level DSLs like CUDA Tile is essential for accelerating AI research and enabling performance engineers to innovate directly within the Python ecosystem.