Running one prompt through an LLM is fairly easy. Keeping a GPU busy while many requests arrive, grow, and finish at different times is a much more interesting systems problem. This talk looks at how vLLM approaches that problem, starting with the memory needed to generate tokens.
I’ve kept every slide from the original architecture presentation below. Each one has a short explanation, and clicking the image opens the full-size diagram. The architecture drawings are simplified views of the implementation discussed in the talk.
1. What makes an inference system fast?
The model is only one part of serving. You also need to decide which requests run together, where their state lives, and when memory can be reused. I wanted to connect those decisions to the GPU work they enable. That’s the thread running through this vLLM walkthrough.
2. Why serving gets expensive
GPU memory holds model weights, temporary workspaces, and the state of active requests. As prompts and generated sequences get longer, request state can become a serious limit on concurrency. The number of requests a GPU can support depends on the model and workload; the broader point is that unused or poorly managed memory has a direct cost.
3. Generating one token at a time
The model processes a prompt, produces a distribution for the next token, and continues using the token selected from that distribution. This repeats until a stopping condition is reached. The growing context matters at every step, but a KV cache lets us reuse earlier attention state instead of recomputing the full prefix each time.
4. What attention needs from the past
For a new token, its query is compared with the keys available in its causal context, and the attention weights combine the corresponding values. That explains why old keys and values remain useful during decoding. Saving them trades memory for avoiding repeated computation, which becomes especially valuable across many generation steps.
5. The transformer has many layers
The diagram places attention alongside the feed-forward computation inside a transformer. In a typical decoder, the attention state is maintained across many layers, so the cache cost isn’t just one set of vectors for the whole model. Layer count, KV head count, head dimension, sequence length, and cache dtype all affect the memory required.
6. Where KV-cache memory gets wasted
If each request reserves a large contiguous region, unknown output lengths make allocation awkward. Some reserved space is never used, some waits for future tokens, and gaps can remain between allocations. The 20–40% figure on the slide describes the earlier systems studied in the PagedAttention paper, rather than a universal utilization number for today’s serving stacks.
7. What vLLM brings together
Paged KV storage is an important starting point, but a serving engine also needs scheduling, efficient kernels, and ways to avoid unnecessary work. Features such as chunked prefill and speculative decoding target different bottlenecks. Which combination helps depends on the request mix, model, and hardware; a feature list alone doesn’t tell us the resulting throughput.
8. What gets cached?
The cache stores key and value vectors from previous tokens at the relevant attention layers. On a normal decode step, we compute the new token’s state and attend over the available cached context. We still perform attention against that history. Caching saves recomputation of earlier K/V projections; it doesn’t make reading a long context free.
9. PagedAttention
A sequence has a logical token order, but its KV blocks don’t have to occupy adjacent physical locations. PagedAttention uses a mapping to find the right blocks while computing attention. This gives the memory manager more freedom to allocate space as a sequence grows, instead of finding one large contiguous reservation at the start.
10. The operating-system analogy
The block table connects each request’s logical blocks to physical KV blocks. Two requests can grow independently, drawing blocks from the same pool, and released blocks can be reused. It’s similar to the separation between virtual and physical memory in an OS, although the serving engine manages these mappings for its own attention execution.
11. The main components
Tokenization turns text into model inputs. The engine coordinates requests, the scheduler selects work, the KV-cache manager tracks storage, and workers execute the model through an executor. Keeping these responsibilities separate helps answer a practical question: is a request waiting for compute, for memory, or for its turn in the schedule?
12. How the pieces connect
Follow a request from the entry point through input processing into the engine core. The scheduler and cache manager prepare work that the executor sends to workers, and output processing turns results back into responses. The host-memory cache drawn here represents an optional configuration path; CPU offloading isn’t a required step for every request.
13. Initialization builds the runtime
Before serving can start, the engine prepares its configuration, workers, model, scheduler, and cache resources. Device initialization and model loading happen on the worker side, while scheduling state is managed by the engine core. The drawing is an approximate initialization map, so individual constructors and process boundaries can differ between versions and deployment modes.
14. Following an actual request
This expanded diagram puts request processing next to scheduling and execution. A request enters the queues, receives a token budget and cache allocation, and participates in model steps until it finishes. The result of one step updates what can run in the next. That repeated feedback is what makes continuous batching useful for requests of different lengths.
15. Beyond the basic execution loop
Once we can schedule requests and manage their KV state, there are several opportunities to improve the service. Long prompts can be processed in pieces, repeated prefixes can reuse work, output formats can constrain sampling, and prefill can run separately from decode. Each feature changes a different part of the request’s journey.
16. Chunked prefill
A long prompt doesn’t have to consume its entire prefill budget in one scheduling step. Splitting it into chunks gives the scheduler room to mix prompt work with ongoing decode requests. The tradeoff is about latency and resource sharing: finishing one prompt as quickly as possible and keeping existing streams responsive are not always the same objective.
17. Prefix caching
If two prompts begin with a compatible, identical prefix, the second request may reuse KV blocks already computed for it. The diagram shows blocks indexed by hashes and shared while needed. Matching requires the relevant execution context as well as token content. This saves repeated prefill work; it doesn’t mean the rest of the response has already been generated.
18. Guided decoding
Before sampling the next token, a structured-output backend can mask tokens that would violate the allowed output structure. The bitmask in the slide illustrates setting invalid logits to negative infinity. A finite-state machine is one useful model, but grammar implementations can be more general. Constraining syntax also doesn’t guarantee that the generated answer is factually correct.
19. Separating prefill and decode
Prefill processes many prompt tokens together, while ordinary decode advances active sequences a small amount at a time. Their resource needs can differ enough to justify separate instances, with KV state transferred between them. That can improve control over time to first token and inter-token latency, but transfer costs and load balancing become part of the design.
20. References for going further
The vLLM codebase connects these diagrams to an implementation, and the PagedAttention paper develops the memory-management argument. The original slides also point to the Anatomy of vLLM article and a talk by Woosuk Kwon and Zhuohan Li. I would start by following one request through the scheduler and engine core.
21. Thanks for reading
I hope the diagrams help connect attention, memory management, and scheduling into one system. If you work on serving, the part I’d be curious to hear about is where your workload runs into a limit first: prompt processing, decoding, or keeping enough requests resident in memory.