Enabling Private High-Performance Production AI Inference with NVIDIA Confidential Computing

NVIDIA Shows Confidential AI Inference on Blackwell with Minimal Performance Hit

NVIDIA has detailed performance results for its Confidential Computing (CC) technology on the new Blackwell GPU architecture, demonstrating that secure, memory-encrypted AI inference is possible with minimal performance trade-offs. Internal testing shows that large language model (LLM) workloads retained over 96% of their baseline throughput when running inside a trusted execution environment. This capability directly addresses a primary barrier for enterprises looking to use generative AI with proprietary models or sensitive data in production settings.

Technical Breakdown of Performance Mitigation

To isolate the impact of its security features, NVIDIA's engineers conducted a controlled comparison running a DeepSeek-R1 model on a DGX B200 system, both with and without Confidential Computing enabled. The results showed a mere 1.2% to 4.3% increase in mean time-per-output-token (TPOT) latency. Achieving this low overhead required co-optimizing the inference framework, NVIDIA TensorRT LLM, to adapt to the secure execution environment. Key framework adaptations included:

• CC-Aware Data Movement: The framework was modified to use pageable memory instead of pinned memory for certain host-to-device transfers, bypassing a bottleneck created by an encrypted bounce buffer.
• Stable Autotuning: TensorRT LLM now uses the GPU’s internal `%globaltimer` instead of CUDA events for kernel timing within a CC environment to ensure the selection of the most efficient tactics.
• Optimized Multi-GPU Communication: The framework now detects the unavailability of NVLink SHARP (NVLS) in CC mode and selects alternative communication paths to avoid unnecessary synchronization costs.

Implications for Enterprise AI

These findings signal that hardware-enforced security for AI is maturing beyond a niche feature into a deployable, high-performance solution. For AI platform engineers, this means security and performance optimization should no longer be treated as separate tasks but as a single, full-stack engineering problem. By substantially reducing the performance penalty for security, NVIDIA is positioning its Blackwell platform to accelerate the adoption of private AI in regulated industries where data confidentiality is non-negotiable.

The minimal performance hit demonstrated by NVIDIA's Confidential Computing stack suggests the long-standing trade-off between security and performance in AI inference is diminishing, removing a significant barrier to enterprise adoption of generative AI for sensitive workloads.