For this PyTorch meetup talk, I started with token generation and worked toward the serving engine around it. The question throughout was simple: what has to happen between receiving a prompt and keeping its response streaming efficiently?
This post follows the meetup deck in its original order, including its transformer refresher. Click any slide to open the diagram at full resolution.
1. Understanding the serving system
vLLM combines model execution with the machinery needed to serve many requests. To understand why that machinery matters, we first need to look at the work a single request creates. We’ll move from tokens and attention to KV-cache allocation, then follow those requests through the engine.
2. The GPU has a memory budget
Loading the model doesn’t leave the remaining memory empty. Every active request brings attention state, and execution needs temporary buffers too. Longer sequences can reduce how many requests fit at once. This is why a serving system’s memory policy affects throughput even when its individual model kernels are already fast.
3. How token generation works
The prompt supplies the initial context. The model predicts a next-token distribution, a token is selected, and generation continues with that token added to the sequence. Requests finish at different points, so a useful scheduler needs to admit new work as old work completes instead of waiting for a whole fixed batch to finish.
4. A quick transformer refresher
The model repeatedly applies attention and feed-forward transformations to token representations, with normalization and residual connections around those computations. The picture helps locate the attention state that we’ll cache. We don’t need every detail of the model architecture to see why retaining that state across decoding steps saves work.
5. Why keys and values matter
Attention lets a token combine information from its available context. During causal decoding, the new token needs access to earlier keys and values at each relevant layer. Those earlier vectors are reusable. That is the starting point for KV caching, and also the reason longer conversations carry a growing memory cost.
6. The problem with large reservations
Output length is unknown when a request arrives. Reserving a contiguous region for its maximum possible length wastes space, while allocating different-sized regions can leave unusable gaps. The utilization range on this slide comes from the historical setting of the PagedAttention work. The allocation problem is the point to carry forward, rather than treating that range as a current benchmark.
7. Several optimizations work together
vLLM addresses more than one source of inefficiency: KV allocation, batching, kernel execution, prompt processing, and token generation. PagedAttention helps organize memory; features such as chunked prefill change how work is scheduled. Their benefits depend on what is actually limiting the service, so it’s useful to understand them individually before combining them.
8. Logical order, flexible storage
PagedAttention lets the KV state for one sequence live in separate physical blocks while preserving its logical order. The attention computation follows the block mapping to access the right state. This means the allocator can grow a request block by block instead of needing a contiguous region large enough for its eventual output.
9. Following the block table
Request A and request B each have their own logical sequence of blocks. Their tables point into a shared pool of physical storage, where blocks can appear in a different order. When a request finishes, its unneeded blocks become available again. The mapping keeps token order separate from allocation decisions.
10. What KV caching actually avoids
The cache preserves earlier key and value projections so decoding doesn’t repeatedly recreate them. New tokens still need their own computation, and attention still reads relevant historical state. That distinction matters: a cache removes redundant work, but a long context can remain expensive to access even when every cached entry is available.
11. Naming the moving parts
The tokenizer handles text representation, the engine coordinates requests, and the scheduler chooses the next work to run. The cache manager supplies storage, while executors and workers carry out model execution. These names become more useful once we connect them to the lifecycle of a request rather than reading them as a flat list.
12. The architecture diagram
Input processing feeds the engine core, which combines scheduling and cache management with model execution. Outputs travel back through processing before reaching the caller. The diagram includes several possible devices and memory arrangements; a particular deployment uses the paths its configuration supports. The main relationship is that scheduling decisions determine the work sent to workers.
13. What initialization prepares
The call graph shows how the engine builds the parts it will need repeatedly: workers, model resources, cache state, and scheduling queues. Initialization can also include profiling and warmup before requests arrive. It’s an approximate map of responsibilities, so I would use it to navigate the source rather than expect every version to preserve the exact constructor sequence.
14. Four features worth understanding
The next slides each change a different part of serving. Chunked prefill shares time between prompt work and decoding. Prefix caching reuses previous prompt computation. Guided decoding constrains what can be sampled. Disaggregated prefill and decode place the two phases on separate serving resources.
15. Processing a long prompt in chunks
Chunking lets a long prefill make progress over multiple scheduling steps. That creates room for active decode requests to keep running instead of waiting behind one large prompt. Choosing the amount of prompt work per step is a latency-throughput tradeoff, which is why the right budget depends on the request mix.
16. Reusing a shared prefix
Many requests may share a system prompt or the start of a document. When compatible cached blocks exist, prefix caching can reuse their KV state and process only the remaining prompt work. The diagram shows why block identity and reference tracking matter: the cache has to recognize reusable computation and keep it alive while requests depend on it.
17. Restricting the next token
Structured output generation applies constraints before sampling. Tokens that would break the current grammar state are masked out, leaving the model to choose among valid continuations. The slide illustrates this with a compact bitmask. This controls output structure, such as valid JSON, while the model still determines the content within those constraints.
18. Moving prefill away from decode
Separating the phases allows a deployment to tune prompt processing and ongoing generation independently. After prefill, the decode instance receives the KV state and continues the sequence. The extra transfer needs to pay for itself, so this is a system-level tradeoff involving network bandwidth, scheduling, and the latency targets of the service.
19. References from the talk
The PagedAttention paper and vLLM source are good places to connect the memory ideas to implementation. The slide also includes the Anatomy of vLLM article, the original technical talk, and the 3Blue1Brown neural-network series used for the visual refresher.
20. Where to go next
Thanks for following the meetup walkthrough. Once the scheduler and cache manager make sense, the next question is how the model’s repeated computation gets optimized. I follow that path in the vLLM and torch.compile post, from the model runner down to compiled graph pieces and CUDA graph replay.