TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor

NVIDIA has established a new performance benchmark for agentic AI on edge devices, demonstrating that its TensorRT Edge-LLM software completed the MLPerf Inference v6.1 Edge Agentic workload 6.4 times faster than a reference implementation. Running the Qwen3.6-27B model on a single NVIDIA Jetson AGX Thor developer kit, the system finished the 1,007-turn task in just 24 minutes and 36 seconds, compared to the 2 hours and 37 minutes required by the llama.cpp reference run. This result is significant as it addresses the growing demand for running complex, multi-step AI agents directly on hardware like vehicles and robots, independent of the cloud.

Technical Breakdown of the Performance Gains

The performance leap was achieved through a combination of software and hardware optimizations tailored for long-context, multi-turn conversations. The submission leveraged several key technologies integrated into the NVIDIA stack, which collectively reduce memory bandwidth requirements and computational overhead. The core components of this accelerated performance include:

• NVFP4 Quantization: A 4-bit floating-point format supported by the Blackwell GPU architecture was used for both model weights and activations, significantly reducing the model's memory footprint.
• Advanced Caching: The system used an FP8 KV cache and, critically, reused the KV cache and recurrent-state across agent turns. This served approximately 96% of prompt tokens from a hot cache, drastically cutting down on redundant prefill computations.
• Tree-Based Multi-Token Prediction (MTP): Instead of generating one token at a time, this technique predicts and verifies multiple potential tokens simultaneously in a tree structure, delivering a reported 40% decoding performance increase over linear MTP for this function-calling workload.

This MLPerf result signals a critical maturation point for edge AI, confirming the viability of deploying sophisticated agents that can reason and use tools locally. By outperforming standard open-source frameworks on its own silicon, NVIDIA is making a clear case for its full-stack ecosystem—from the Jetson AGX Thor hardware to the highly optimized TensorRT Edge-LLM runtime. For developers and enterprises, this integrated approach promises lower latency, enhanced privacy, and greater autonomy for AI applications operating outside the data center, potentially accelerating adoption in robotics, industrial automation, and autonomous vehicles.

NVIDIA's MLPerf result is less about raw token speed and more a strategic demonstration of how its full-stack optimizations—from the Blackwell GPU's NVFP4 support to TensorRT Edge-LLM's advanced caching—create a performance moat that open-source software alone struggles to cross on its hardware, cementing its position in the high-value edge agent market.