10 min read
LLM Inference at Scale: Why Serving One User Is Easy and Serving a Hundred Is Not
A ground-up walk through LLM inference: the KV cache, prefill vs decode, a capacity equation for Mistral 7B on an A100, and the model and serving optimizations (GQA, MLA, FlashAttention, PagedAttention, continuous batching, prefix caching, speculative decoding) that make LLMs servable.
LLMInferencevLLMSGLangKV CacheSystems