Benchmarking LLM Inference at Scale with AIPerf

NVIDIA Releases AIPerf for Scalable LLM Benchmarking

NVIDIA has launched AIPerf, a new tool for benchmarking large language model (LLM) inference performance, designed as a successor to the earlier GenAI-Perf. The tool directly addresses a common failure point in performance testing where the load-generating client itself becomes a bottleneck, leading to unreliable metrics. AIPerf's multiprocess architecture is engineered to properly saturate high-concurrency inference servers, providing engineers with a more accurate and trustworthy measure of how a system will perform under production-level stress.

• Multiprocess Architecture: Uses separate worker and record-processor services to prevent client-side bottlenecks, a limitation of its single-process predecessor.
• Realistic Workloads: Supports over 15 endpoint types and can replay production traffic from trace formats including Mooncake, Baseten, and WEKA AgentX.
• Configurable Traffic Shaping: Allows engineers to model traffic using constant, Poisson, and gamma arrival patterns to mimic real-world request distributions.
• Core Metrics Reporting: Measures Time to First Token (TTFT), Inter-Token Latency (ITL), request latency, and throughput with detailed percentile breakdowns.
• GPU Telemetry: Integrates with NVIDIA DCGM or pynvml to correlate performance metrics with GPU power draw, utilization, and memory consumption.

Unlike single-process tools that are often limited by Python's Global Interpreter Lock (GIL), AIPerf is a ground-up rewrite that distributes work across multiple processes coordinated over ZMQ. This allows it to generate sufficient load to identify the true performance limits of an inference server. Developers can configure synthetic workloads with variable input and output lengths or use public datasets like ShareGPT to simulate diverse and realistic scenarios. This gives operators control over not just the volume of traffic but its specific statistical shape.

The introduction of AIPerf provides the AI industry with a standardized method for a previously fragmented process. Instead of relying on ad-hoc scripts or homegrown load generators, teams can now use a consistent and reproducible tool for capacity planning, comparing different model serving stacks, and validating service-level objectives. By providing a client capable of out-pacing the server, NVIDIA is enabling a more rigorous and honest evaluation of inference infrastructure before it is deployed to production.

NVIDIA's AIPerf is a direct response to the 'garbage in, garbage out' problem of LLM benchmarking. By engineering a client that can overwhelm nearly any server, they are shifting the focus from the limitations of the testing tool to the actual performance characteristics of the model and serving infrastructure. It standardizes the measurement process, forcing a more honest evaluation of inference-at-scale.