llama.cpp brings LLM inference to the hardware you already have

01Plain C and C++ on top of ggml

The README states the goal as LLM and VLM inference with minimal setup and what it calls state-of-the-art performance on a wide range of hardware, locally and in the cloud. The implementation is plain C and C++ without dependencies, built on the ggml tensor library. Apple Silicon is a first-class target through ARM NEON, Accelerate and Metal; x86 gets AVX, AVX2, AVX512 and AMX paths; RISC-V vector extensions are supported; and hybrid CPU and GPU inference can accelerate models larger than the available VRAM.

The build guide explains that the main product is the llama library, with a C-style interface in include/llama.h, and that the repository also holds programs ranging from minimal examples to the HTTP server. That split is useful. Applications can link the library directly, call the server over HTTP or rely on a project that embeds the engine; the Open WebUI README, for one, describes a desktop app with a built-in llama.cpp engine for fully local inference.

02Quantization is the main lever

llama.cpp supports 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit and 8-bit integer quantization for faster inference and lower memory use. Models come in the GGUF format, and the README's own examples download them straight from Hugging Face with the -hf flag. A smaller quantization lets a larger model fit on a given machine or leaves room for a longer context, at a cost in output quality that varies by model and task.

Choose the quantization by evaluation, not by file size alone. Run the same prompts through two or three variants of a model, compare answers on the tasks that matter, then measure memory and tokens per second on the real target machine. Record the exact GGUF file and where it came from, because two files with similar names can come from different conversions. The model's own license still applies to quantized copies, so the license check travels with the file.

03A long list of backends

The supported backends table includes Metal for Apple Silicon, CUDA for NVIDIA, HIP for AMD, MUSA for Moore Threads, Vulkan and WebGPU for general GPUs, SYCL for Intel GPUs, CANN for Ascend NPUs, OpenCL for Adreno, Hexagon for Snapdragon, ZenDNN for AMD CPUs, IBM zDNN, BLAS and BLIS, and an RPC backend for spreading work across machines, with OpenVINO marked as in progress. The build guide gives a plain CPU build as two CMake commands: cmake -B build, then cmake --build build --config Release.

Pre-built binaries and the install script cover common platforms; other backends need a source build with the matching SDK. Treat the backend as part of the deployment's identity. A model that runs well on Metal may behave differently on Vulkan, and a driver update can change speed or stability. Keep a short benchmark and a known-good build for each machine type, and rerun it after driver or toolkit upgrades.

04The server has grown into an application host

llama-server, built on cpp-httplib and nlohmann::json, offers OpenAI-compatible chat completions, responses and embeddings routes, an Anthropic Messages-compatible endpoint, reranking, multimodal input, function calling for almost any model, schema-constrained JSON output, speculative decoding, continuous batching for several users, monitoring endpoints and a web UI. Prometheus-compatible metrics are served at /metrics only when the server starts with --metrics, and a router mode serves more than one model.

One binary can stand in for a hosted API during development or in a small private deployment, which is convenient. It also means the server's flags deserve the same review as any service configuration. The slots monitoring endpoint is on by default and can be turned off with --no-slots, and changing global properties through POST /props requires --props. Decide which endpoints a deployment needs, enable metrics on purpose, and keep the full command line under version control.

05Tools and MCP give the model real access

Recent server versions can give the model built-in tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file and get_info. They are off by default, enabled with --tools, and the help text says not to enable them in untrusted environments. MCP servers declared in a Cursor-compatible JSON file add further tools; each runs as a child process with the same privileges as the server, and the README says to declare only commands you trust. The --agent flag turns on a CORS proxy and all built-in tools together.

When any of these are enabled, --cors-origins defaults to localhost, so only pages served from the local machine can reach the server. Without them, the server reflects any Origin header back with credentials allowed, which the docs describe as acceptable only while the server exposes stateless, read-only endpoints. The working rule is to keep tool access to one user on one machine, and to run anything broader inside a sandbox where shell and file access cannot reach data that matters.

06Network exposure and authentication

The server listens on 127.0.0.1 by default, and since v0.7.0 its default port is 9931 instead of 8080, a change the release notes call out. An API key can be set with --api-key or LLAMA_API_KEY, several keys can be given as a comma-separated list, and --api-key-file reads keys from a file. TLS requires a build with OpenSSL and the --ssl-key-file and --ssl-cert-file options. For public deployments, the server README's CORS table recommends an API key and a reverse proxy.

Port changes like this one break scripts, health checks and container mappings without much noise, so pin the build and pass --port explicitly instead of relying on a default. The Docker examples in the server README bind 0.0.0.0 inside the container and publish port 8080, so check what the host side exposes. A model endpoint with no key on a reachable interface is free compute for anyone who finds it, and with tools enabled it is considerably worse.

• Set --host and --port explicitly in every service definition.
• Use --api-key-file so keys stay out of process listings and shell history.
• Put TLS and access logging on a reverse proxy when the server must be reachable from other machines.

07Two release channels

The README's badges point to two kinds of release: versioned releases tagged v, and nightly builds tagged b. The GitHub releases API listed v0.7.0, published October 15, 2026, as the latest release when this guide was checked. Its notes describe a GPU cache for mixture-of-experts weights held in host memory, new speculative decoding methods, a batch of newly supported model architectures, a refactored chat session API in the server, the new default port and an update of the bundled ggml to v0.27.0.

The nightly channel suits people following new model support closely; the versioned channel suits deployments that want fewer surprises. Pick one per environment and record the exact tag. The release notes list API changes to include/llama.h, which matters to anyone linking the library rather than calling the server. Test a new build with the same prompts and the same GGUF files before switching, and keep the previous binary until the new one has run for a while.

08Choosing llama.cpp or something around it

The Ollama guide on this site covers a runtime that wraps local models in a model library and a small API. The vLLM guide covers high-throughput serving on server GPUs. llama.cpp offers the widest hardware reach of the three, direct control over quantization and flags, and a server that now covers most of what an application needs from a model endpoint. It is MIT licensed, which keeps embedding it in other software simple.

Use llama.cpp directly when the hardware is unusual, when the exact quantization and flags matter, or when one self-contained binary is easier to ship than a Python stack. Use a higher-level tool when a model library and simpler defaults matter more than control. Either way, keep the model file, the build tag and the command line together in the deployment record, since those three decide what the endpoint actually does.