Transformers now runs llama.cpp quants

Hugging Face Integrates GGUF Models into Transformers Library

Hugging Face announced it has integrated support for the GGUF model format directly into its popular `transformers` library, enabling developers to run highly efficient, quantized models popularized by llama.cpp. This update simplifies using compact models on local hardware, such as laptops with Apple Silicon, through the familiar PyTorch-based `transformers` API. The move bridges the gap between the high-performance C++ inference world of llama.cpp and the flexible Python development environment, allowing the same quantized checkpoints to be used for both deployment and experimentation.

Technical Details and Performance

To achieve performance comparable to native llama.cpp, the integration reuses the underlying `ggml` Metal kernels, which are specialized GPU programs for operations on quantized weights. The initial implementation focuses on Apple Silicon Macs, starting with the Qwen3.5 architecture. Loading a GGUF model requires only a few lines of code using the `from_pretrained` function, and these models can also be served through an OpenAI-compatible API using `transformers serve`. Benchmarks indicate that the `transformers` implementation is close in performance to llama.cpp, although the measurements differ slightly, with the `transformers` test including the prefill stage.

• Unified Workflow: Use the same GGUF files in `transformers` that are used in tools like Ollama and LM Studio.
• Familiar APIs: Leverage standard `transformers` and PyTorch tools for generation, evaluation, and even fine-tuning by dequantizing models.
• Hardware Focus: The first release prioritizes efficient inference on Apple Silicon by using `ggml`'s Metal kernels.
• Quantization Support: Users can select from various quantization levels, such as the popular Q4_K_M, to balance model quality with memory footprint.

Ecosystem Impact and Future Direction

This integration is not intended to replace llama.cpp as a dedicated, high-performance inference engine. Instead, it offers developers a convenient pathway to inspect, evaluate, and prototype with GGUF models within a PyTorch environment. The long-term potential extends beyond the GGUF format itself; by making `ggml` kernels available in PyTorch, Hugging Face can accelerate operations for new and experimental model architectures without needing a full C++ implementation for each one. This could significantly speed up research and development for models that may never get a dedicated llama.cpp port, including those in other modalities like vision and audio.

By integrating GGUF support, Hugging Face is strategically reducing ecosystem fragmentation. This move solidifies `transformers` as the central framework for the entire model lifecycle, from research and training in PyTorch to efficient, quantized deployment on local devices, thereby capturing developers who might otherwise operate solely within the llama.cpp ecosystem.