vLLM makes GPU memory the center of LLM serving

01Throughput is a memory problem first

The vllm-project/vllm README describes a library for LLM inference and serving, and its list of strengths starts with memory. PagedAttention is the project's approach to managing attention key and value memory, and it is the subject of the 2023 SOSP paper the README asks researchers to cite. Continuous batching admits incoming requests as others finish, while chunked prefill and prefix caching cut repeated work on long or shared prompts. These features, more than raw kernel speed, set how many conversations one GPU can carry.

That focus explains both the appeal and the learning curve. A model that loads successfully can still serve badly if little memory is left for the cache after the weights, because every concurrent request competes for that space. Throughput figures in release notes come from specific models, GPUs and prompt shapes. The useful habit is to plan the whole deployment around memory: weight precision, context length, the number of concurrent sequences and the cache all trade against each other, so changing one without measuring the others is how a working server starts queueing or rejecting requests.

02Two ways in: offline batches and an API server

The quickstart presents two entry points. Offline batched inference uses the LLM class and SamplingParams from Python, which suits evaluation runs, dataset labeling and other jobs where one script owns the whole queue. Online serving starts with vllm serve and a model name, and exposes an HTTP server that implements the OpenAI API protocol on port 8000 of localhost by default. The docs point out that llm.generate does not apply a chat template, so instruct and chat models need llm.chat or a template applied by hand.

Defaults deserve a careful read. The server applies the model repository's generation_config.json when one exists, so sampling values can come from the model author rather than from vLLM, and --generation-config vllm turns that off. The quickstart also states that the server hosts one model at a time. An application that expects several models behind one URL needs several processes and a router in front of them. The README adds that current versions also accept the Anthropic Messages API and gRPC, which widens the client choice without changing those operating rules.

03Installation is a hardware decision

For NVIDIA GPUs the quickstart recommends uv: create a Python 3.12 environment, install vllm with --torch-backend=auto so uv selects the PyTorch index that matches the installed CUDA driver, then run vllm download-kernels. That last step installs precompiled kernels that vLLM would otherwise download and compile at startup, which the docs say can add several minutes on Hopper and newer GPUs. The prerequisites list Linux and Python 3.11 through 3.14.

Other accelerators take other routes. AMD users install from a ROCm wheel index, Google TPUs use a separate vllm-tpu package, Ascend NPUs go through the community vLLM Ascend plugin, and Apple Silicon runs through vLLM-Metal, which uses MLX instead of PyTorch and needs MLX-optimized models. Each path has its own versions and caveats. A team with mixed hardware should treat every target as a separately supported platform with its own image, test run and rollback, rather than assume one requirements file covers them all.

04Parallelism and quantization widen what fits

When a model is too large for one device, the README lists tensor, pipeline, data, expert and context parallelism for distributed inference. Quantization options run from FP8 and INT8 down to 4-bit formats and GGUF, and attention and matrix kernels come from several projects, including FlashAttention, FlashInfer and CUTLASS. Speculative decoding, structured outputs through xgrammar or guidance, tool calling and reasoning parsers, and multi-LoRA serving are also on the list. The menu is long because the project supports many combinations of hardware and model families.

Each option is a measurement, not a free upgrade. A lower-precision format changes output quality in ways that depend on the model and the task, so run the application's own evaluation set before and after. Speculative decoding helps some prompt shapes and does little for others. Multi-node parallelism puts a network on the critical path. Start with the smallest configuration that meets the latency target, record the exact flags, and add one technique at a time so that a regression can be traced to the change that caused it.

• Measure time to first token and tokens per second with the longest realistic prompts, not a short greeting.
• Record model revision, precision, context length and every serving flag together in the deployment record.
• Keep a fixed prompt set to replay after any change of model, quantization or vLLM version.

05Security defaults assume a private network

vLLM's security guide is plain about multi-node deployments: communication between nodes is insecure by default and has to be protected by placing the nodes on an isolated network. That covers PyTorch Distributed traffic, KV cache transfer and the tensor, pipeline and data parallel channels. The guide recommends setting VLLM_HOST_IP to a specific address instead of relying on defaults, allowing only the needed ports between nodes and restricting physical and network access. The KV transfer and data parallel master addresses default to 127.0.0.1 with fixed ports.

The HTTP side needs its own policy. API key checking is enabled only with --api-key or the VLLM_API_KEY variable, and several keys can be passed to support rotation. For multimodal inputs, the guide recommends --allowed-media-domains so the server cannot be used to fetch arbitrary URLs, including cloud metadata addresses, and it documents size limits such as VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB, which defaults to 256. The v0.32.0 release notes add that CORS no longer allows every origin by default and that untrusted images are limited to raster formats.

06Release cadence is part of the operating cost

The GitHub releases API listed v0.32.0, published on October 18, 2026, as the latest release when this guide was checked. Its notes count 814 commits from 363 contributors and cover new endpoints, kernel work and model support, followed by a list of breaking changes: Python 3.10 was dropped, the supported transformers and tokenizers versions were narrowed, cascade attention and some compilation passes were removed, and max_tokens is no longer reserved from the context window. In this project a minor version number does not mean a small change.

Plan upgrades the way a team plans a database migration. Pin vLLM and the PyTorch backend in the serving image, keep the previous image available, and replay a fixed set of prompts through both before moving traffic. Watch changed defaults as closely as new features, because a default that moves, such as CORS or context reservation, changes behavior without any client code changing. Security fixes ship in the same release stream, which is another reason not to fall several versions behind.

07Where vLLM fits beside lighter runtimes

Elsewhere on this site, the Ollama guide covers making one local model feel like an ordinary service, and the llama.cpp guide covers a C and C++ engine aimed at minimal setup on a wide range of hardware. vLLM sits at the other end: many simultaneous requests on server GPUs, distributed inference and a scheduler designed around concurrency. It fits when cost per token and requests per second are the numbers that matter, and when someone on the team is ready to own the GPU host.

It is a weaker fit for a single developer on a laptop, for a team without access to a supported accelerator, or for a product that needs many small models loaded and unloaded on demand behind one endpoint. In those cases a lighter runtime or a hosted API carries less operational weight. The practical test is load: if the workload rarely has more than a few requests in flight, the batching machinery that makes vLLM valuable has very little to do.

08The license covers the engine, not the weights

The repository is released under the Apache License 2.0, which allows commercial use and modification subject to its notice and attribution conditions. That covers the serving engine. Every model served through it carries its own license and acceptable-use terms, and those differ widely; some impose conditions on commercial use, redistribution or naming. vLLM downloads models from Hugging Face by default, so the model card and license file on the model repository are the documents to check for each deployment.

Keep a record per deployed model: the exact repository and revision, its license text, any quantization applied, the vLLM version and the flags used to serve it. That record makes an audit possible and rollbacks predictable. It also helps when a model publisher updates a checkpoint in place, because the deployment can be compared with what was actually evaluated instead of with a model name that now points somewhere new.