llama.cpp

llama.cpp is a plain C/C++ implementation of LLM inference developed by Georgi Gerganov and maintained by the ggml-org community. It is designed to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware — from consumer laptops to enterprise GPUs — running locally and in the cloud.

Unlike many inference frameworks that depend on Python and PyTorch, llama.cpp has zero required dependencies beyond a C++ compiler. It supports Apple Silicon as a first-class citizen via Metal, x86 architectures with AVX/AVX2/AVX512/AMX, RISC-V, NVIDIA GPUs via CUDA, AMD GPUs via HIP, Intel GPUs via SYCL and Vulkan, and more. The project is MIT-licensed and hosted on GitHub at github.com/ggml-org/llama.cpp.

llama.cpp introduced the GGUF format, which has become the de facto standard for distributing quantized LLMs. The project powers most local LLM tools in the ecosystem, including Ollama, LM Studio, LocalAI, and hundreds of other applications. It also introduced the llama-server binary that exposes an OpenAI-compatible REST API for production deployments.

llama.cpp can be installed through package managers or built from source. Pre-built binaries are available on the GitHub releases page.

Package Managers

Manager	Command
Homebrew	brew install llama.cpp
Nix	nix-shell -p llama-cpp
Winget	winget install llama.cpp
Conda	conda install -c conda-forge llama-cpp

Build from Source

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release

Enable GPU backends with CMake flags like -DLLAMA_CUDA=ON, -DLLAMA_METAL=ON, or -DLLAMA_VULKAN=ON.

Docker

docker run -it --rm \
  -v /path/to/models:/models \
  ghcr.io/ggml-org/llama.cpp:full-cuda

llama.cpp supports an extensive range of model architectures, including both text-only and multimodal models. Text-only support includes LLaMA 1/2/3, Mistral, Mixtral, Falcon, Gemma, Qwen, DeepSeek, Yi, Phi, GPT-2, Mamba, and hundreds of fine-tuned variants. Multimodal support includes LLaVA 1.5/1.6, BakLLaVA, Qwen2-VL, Moondream, MiniCPM, and more.

Category	Example Models
Text LLMs	LLaMA 3, Mistral, Mixtral, Gemma, Qwen 2.5, DeepSeek, Phi-4, Falcon
Multimodal	LLaVA 1.6, Qwen2-VL, Moondream, MiniCPM-V, LLaMA 3.2 Vision
Code Models	CodeLLaMA, DeepSeek Coder, StarCoder, Qwen2.5-Coder
MoE Models	Mixtral 8x7B, Qwen2 MoE, DBRX, DeepSeek V2/V3

Models are typically distributed in GGUF format and can be downloaded directly from Hugging Face Hub using the -hf flag: llama-cli -hf ggml-org/gemma-3-1b-it-GGUF.

llama.cpp pioneered the GGUF (GGML Universal Format) file format, which packages model weights, tokenizer, and metadata into a single self-contained file. GGUF supports a wide range of quantization levels that trade precision for reduced memory usage and faster inference.

Quantization	Bits per Weight	Use Case
Q2_K	2.6	Maximum compression, lowest quality
Q3_K_M / Q3_K_L	3.5	Very compressed, usable quality
Q4_K_M / Q4_K_S	4.5	Recommended balance of quality and size
Q5_K_M / Q5_K_S	5.5	Higher quality, larger file size
Q6_K / Q8_0	6.6 / 8.5	Near-lossless quality
F16	16	Full precision, maximum quality

llama.cpp provides a pluggable backend system that targets different hardware. The Metal backend delivers native performance on Apple Silicon, CUDA supports NVIDIA GPUs with custom kernels, HIP enables AMD GPU acceleration, Vulkan provides cross-platform GPU compute, and SYCL targets Intel GPUs. CPU-only operation is fully supported with BLAS, BLIS, and ZenDNN acceleration. The RPC backend allows distributed inference across multiple machines.

• Apple Silicon — Metal, Accelerate, ARM NEON
• NVIDIA — CUDA with custom kernels, Tensor Core support
• AMD — HIP (ROCm), Vulkan
• Intel — SYCL, OpenVINO, Vulkan
• CPU — AVX2, AVX512, BLAS, BLIS, ZenDNN
• RISC-V — RVV, ZVFH support
• Web — WebGPU for browser-based inference

CPU+GPU hybrid mode allows partial acceleration of models larger than total VRAM, splitting layers between GPU and CPU automatically.

The llama-server binary provides an OpenAI-compatible REST API, exposing chat completions, completions, embeddings, reranking, and tokenization endpoints. It supports multimodal inputs (images + text) and includes a built-in web UI for interactive use.

llama-server -m model.gguf --port 8080 --host 0.0.0.0

API features include streaming (server-sent events), JSON mode, grammar-constrained generation, tool calling / function calling, vision support, and batching. The server also exposes Prometheus metrics for monitoring.

CLI Tools

Command	Purpose
llama-cli	Interactive chat or one-shot inference
llama-server	OpenAI-compatible HTTP API server
llama-quantize	Quantize models to lower bit depths
llama-perplexity	Evaluate model perplexity on a dataset
llama-embedding	Generate text embeddings
llama-bench	Benchmark inference performance
llama-convert	Convert models to GGUF format

llama.cpp has bindings for virtually every major programming language. Python developers commonly use llama-cpp-python by abetlen, which provides a full Python API with OpenAI-compatible server support. Node.js users can use node-llama-cpp, and Go developers have go-llama.cpp. Other bindings exist for Rust, C#, Java, Ruby, Swift, Zig, Dart/Flutter, and PHP.

The ecosystem also includes popular UIs and tools built on llama.cpp: Ollama uses llama.cpp as its inference backend, LM Studio and LocalAI integrate it, and dozens of applications from GPT4All to Jan and KoboldCPP depend on it. The llama.vim and llama.vscode plugins bring local FIM code completions to editors.