Hugging Face Transformers Adds Native llama.cpp GGUF Model Support
Summary
- • HuggingFace Transformers now loads GGUF quantized models natively via llama.cpp kernels
- • Apple Silicon Macs first supported; Qwen3.5 architecture is the first supported model family
- • Simple API: pass gguf_file to from_pretrained — rest is standard Transformers workflow
- • GGUF variants from Q4_K_M (2.74 GB) to BF16 (8.42 GB) offer flexible size-performance tradeoffs
Details
Native GGUF/llama.cpp support added to Transformers
HuggingFace Transformers library now natively loads and runs llama.cpp GGUF quantized models, without requiring separate inference tools like Ollama or LM Studio.
GGUF format: weights, metadata, tokenizer in one file
GGUF packages model weights, metadata, and tokenizer in a single file and supports quantization levels from BF16 (full precision) to Q4_K_M (4-bit mixed precision).
Performance via native ggml kernels and ggml-attn
Integration uses llama.cpp's underlying ggml kernels through the `kernels` library and ggml-org/ggml-attn for attention, achieving near-native llama.cpp inference speed.
Initial platform limited to Apple Silicon Macs
First supported platform is Apple Silicon; when ggml kernels are unavailable the model falls back to standard sdpa attention with a warning, or sdpa can be forced explicitly.
Qwen3.5-4B: 8.42 GB (BF16) down to 2.74 GB (Q4_K_M)
Quantization dramatically reduces file size: Q4_K_M is 2.74 GB, Q5_K_M 3.14 GB, Q6_K 3.53 GB vs 8.42 GB for the unquantized BF16 reference.
GGUF models downloaded millions of times on HF Hub
GGUF models have accumulated millions of downloads on HuggingFace Hub; major publishers include Unsloth, bartowski, and LM Studio Community.
Technical details of Hugging Face Transformers' native llama.cpp GGUF model support launch.
What This Means
Hugging Face Transformers integrating llama.cpp GGUF support removes a major friction point in the local AI ecosystem: developers no longer have to choose between the familiar Transformers API and llama.cpp's quantization performance. With GGUF models already downloaded millions of times, this integration lets a huge existing user base run efficiently quantized models on consumer hardware using the same workflow they already know. Starting with Apple Silicon and Qwen3.5 narrows the initial audience, but the architecture is clearly designed to expand, and the precedent of Transformers as a unified local inference front-end could accelerate local AI adoption significantly.
Sources
- Transformers now runs llama.cpp quantsHugging Face