LLM Inference at Scale: Why Serving One User Is Easy and Serving a Hundred Is Not

These started as handwritten notes while I was going deep on LLM serving. This post is the cleaned-up version: what actually happens when a model generates a token, where the memory goes, and why tools like vLLM and SGLang exist.

The problem

Running a 7B model for yourself on a single GPU feels fast. Put that same model behind an API with real traffic and three problems show up almost immediately:

1. Memory grows with every token. Each request holds state that grows linearly with its context length.
2. Time to first token (TTFT) gets slow as prompts get longer. A 16K-token prompt has to be fully processed before the first output token appears.
3. Latency degrades as concurrency rises. As the queue fills, every user waits longer, and eventually the GPU simply runs out of memory.

All three come back to one data structure: the KV cache. To see why, we need to look at what happens inside the model for each token.

The inference pipeline

• Tokenizer turns text into integer token IDs.
• Embedding maps each ID to a vector.
• Transformer layers (32 of them in Mistral 7B) do the actual work.
• LM head projects the final hidden state to a probability over the vocabulary, and we sample the next token.

Inference has two phases. Prefill processes the prompt tokens in parallel, builds their KV cache, and produces the first output token. Decode generates subsequent tokens sequentially, reusing and extending that cache. A 500-token answer starts with prefill and continues through sequential decode steps. The transformer layers account for the overwhelming majority of the compute, so that is where we look next.

Inside a transformer layer

Input --> RMSNorm --> Attention --> (+) --> h
  |                                  ^
  +----------------------------------+
 
h --> RMSNorm --> Feed-forward --> (+) --> Output
|                                   ^
+-----------------------------------+

Each layer has two blocks: attention and a feed-forward network, each wrapped with normalization and a residual connection. The feed-forward block treats every token independently. Attention is the part that lets tokens look at each other, and it is the part that creates the memory problem.

Attention, step by step

Step 1: every token becomes Q, K and V. Each token's hidden state is projected into three vectors:

• Query (Q): what this token is looking for.
• Key (K): the label this token advertises to others.
• Value (V): the information this token carries.

Step 2: score. The current token's Q is dotted against the K of the current token and all preceding tokens. Dividing by √d keeps the scores in a stable range, and softmax turns them into weights that sum to 1.

Step 3: mix. The output is the weighted sum of the V vectors of the current token and all preceding tokens.

Here is the important part: when processing token n, the model needs its own K and V and those of all n − 1 preceding tokens, in every layer. The preceding tokens' K and V vectors never change once computed. So instead of recomputing them on every step, we compute them once and store them. That store is the KV cache.

How big is the KV cache?

For Mistral 7B (32 layers, 8 KV heads, head dimension 128, FP16 = 2 bytes):

KV per token = 2 (K and V) × 32 layers × 8 heads × 128 dims × 2 bytes
             = 131,072 bytes ≈ 131 KB per token

That sounds small until you multiply it out:

The model weights are a fixed cost. The KV cache is the variable cost, and it scales with users × context length.

The GPU memory budget

Mistral 7B has about 7.24B parameters. At 2 bytes each in FP16, that is ~14.5 GB of weights. Add roughly 8 GB for activations, CUDA context and framework overhead. Everything else on the card is room for KV cache, and that room is what decides how many people you can serve.

Two phases of inference

Every request goes through two very different phases.

Prefill

The whole input prompt is processed in one forward pass. All prompt tokens are known up front, so the work runs in parallel as large matrix multiplications, and the KV cache for the entire prompt is built in this pass.

• Compute-bound: the GPU's arithmetic units are the limit.
• Measured by TTFT (time to first token). Longer prompts mean longer prefill, which is why TTFT grows with context size.

Decode

The model generates output tokens one at a time, sequentially. Each step processes just one new token, appends its K and V to the cache, and reads the entire cache back to compute attention.

• Memory-bandwidth-bound: each step does very little math but has to stream all the weights plus the request's whole KV cache out of GPU memory.
• Measured by TPOT (time per output token).

Decode is the bottleneck. It is sequential by nature, it underuses the GPU's compute, and its cost per step grows with the cache. Most serving optimizations are, one way or another, about making decode cheaper or doing more of it at once.

The capacity equation

Putting the pieces together:

available KV memory = GPU memory − weights − overhead
max concurrent users = available KV memory ÷ (KV per token × context length)

For Mistral 7B on a single A100-80GB with 4K context:

available = 80 − 14.5 − 8 = 57.5 GB
per user  = 131 KB × 4,096 ≈ 537 MB
max users = 57.5 GB ÷ 537 MB ≈ 107 concurrent users

At 16K context, the same GPU fits only about 26 users. Context length directly divides your capacity.

The trade-off triangle

You get to optimize for two of these three at a time:

• Quality + Latency: big model, small batches, fast responses. Expensive per user. Think premium chat.
• Quality + Throughput: big model, large batches, and users wait. Fine for offline batch jobs and asynchronous agents.
• Latency + Throughput: smaller or heavily quantized models served at high concurrency. Think autocomplete or a free tier.

Every optimization below either moves you along one edge or pushes the whole triangle outward.

Model-level optimizations

These change the model itself or how its math runs.

Weight quantization

Store weights in fewer bits: FP16 → INT8 → INT4. Mistral 7B drops from ~14.5 GB to roughly 7 GB at INT8 and under 4 GB at INT4. Every gigabyte freed from weights becomes KV cache, which means more concurrent users. It also speeds up decode, since there are fewer bytes to stream per step. The cost is some quality loss, which grows as you go lower.

Grouped-Query Attention (GQA)

In standard multi-head attention (MHA), every query head has its own K and V head. GQA lets groups of query heads share one K/V head.

Example: 8 query heads split into 2 groups. Heads 1–4 share one set of K/V, heads 5–8 share another. The KV cache shrinks 4×.

Mistral 7B already does this: 32 query heads share 8 KV heads. That is why its KV cost is 131 KB per token rather than ~524 KB. Llama and Qwen models use GQA too.

Multi-Head Latent Attention (MLA)

Introduced with DeepSeek-V2, MLA goes further. It compresses keys and values into a shared low-rank latent vector and caches only that compact latent, reconstructing per-head K and V when needed. The result is a much smaller KV cache while keeping quality close to full MHA.

FlashAttention

FlashAttention doesn't change what attention computes. It is exact. It changes how the computation runs on the GPU.

A GPU has large but slow HBM (the 80 GB) and tiny but very fast on-chip SRAM. Naive attention writes the full n × n score matrix out to HBM and reads it back. FlashAttention splits Q, K and V into tiles that fit in SRAM, computes attention tile by tile with a running softmax, and never materializes the full matrix. Less data movement means faster attention and much lower memory use for long sequences.

Serving-level optimizations

These live in the inference engine, not the model.

The KV cache itself

The cache is the first and biggest optimization. Without it, each decode step would recompute K and V for every previous token: O(n) work per step and O(n²) over a whole generation. With it, each step computes K and V only for the new token, and total projection work becomes O(n). The cache trades memory for compute, which is exactly why memory became the constraint.

PagedAttention

Inspired by virtual memory in operating systems.

Early serving systems reserved one contiguous chunk of memory per request, sized for the maximum possible output. Most of it sat unused, and fragmentation wasted more. The vLLM paper measured existing systems using as little as ~20% of their KV memory for actual tokens.

PagedAttention splits the KV cache into small fixed-size blocks (pages), allocates them on demand, and keeps a block table per request, just like an OS page table. Waste drops to a few percent, and the vLLM authors report 2–4× higher throughput, largely from fitting far more concurrent requests on the same GPU.

Continuous batching

Static batching waits for every request in a batch to finish before starting the next batch. One long response holds everyone else up.

Continuous batching schedules at the individual iteration (token) level. After every decode step, finished requests leave the batch and waiting requests join immediately. The GPU stays full and short requests are not stuck behind long ones.

Prefix caching

In a multi-turn conversation, every new turn resends the whole prior conversation. Many requests also share the same system prompt. Prefix caching keeps the KV cache for prompts it has already seen, so a new request only needs prefill for the part that is actually new. This cuts both compute and TTFT.

KV cache quantization

Store the cache in FP8 instead of FP16. That halves KV memory and roughly doubles the number of tokens you can hold, typically with negligible accuracy loss.

PagedAttention, continuous batching, prefix caching and FP8 KV cache are all available in vLLM today.

Speculative decoding

Decode is slow because it is sequential, one token per forward pass of the big model. Speculative decoding uses two models:

• a small, fast draft model that quickly guesses the next several tokens, and
• the large target model, which checks the whole guessed sequence in one parallel forward pass.

The target accepts the longest correct prefix of the guesses and corrects the first wrong token. When the draft is usually right, you get several tokens for roughly the cost of one big-model step, and with the standard acceptance rule the output distribution is identical to the target model alone.

RadixAttention

SGLang generalizes prefix caching with RadixAttention. It keeps cached KV for all requests in a radix tree keyed by token sequences. When a new prompt arrives, the system walks down the tree to find the longest matching token prefix already in memory, reuses it, and only computes the rest. Least-recently-used branches are evicted when memory gets tight.

This pays off whenever requests share structure: common system prompts, few-shot examples, multi-turn chat, and agent loops that keep re-sending the same scaffolding. The SGLang authors report up to ~5× higher throughput than earlier systems on these prefix-heavy workloads.

Takeaways

• Prefill processes prompt tokens in parallel and produces the first output token; decode generates subsequent tokens sequentially.
• Attention needs the K and V of the current token and all preceding tokens, so we cache them. That KV cache is the main consumer of serving memory, and it scales with users × context.
• Capacity is a budget: (GPU − weights − overhead) ÷ (KV per token × context). For Mistral 7B on an A100-80GB at 4K context, that is ~107 users.
• Prefill is compute-bound and sets TTFT. Decode is memory-bound, sets TPOT, and is the real bottleneck.
• Model-side fixes shrink what you store (quantization, GQA, MLA) or move it more efficiently (FlashAttention).
• Serving-side fixes waste less memory (PagedAttention), keep the GPU busy (continuous batching), avoid repeated work (prefix caching, RadixAttention), store it smaller (FP8 KV), or break the one-token-at-a-time limit (speculative decoding).

Next up, I want to put real numbers behind these by benchmarking vLLM and SGLang side by side on prefix-heavy and long-context workloads.