← Reference · Nestor G Pestelos Jr · Print this page

Reference Document

LLM Inference

A citable reference on Large Language Model inference: the two-phase prefill-decode lifecycle, operational bottlenecks, KV cache management, and high-throughput serving systems.

1. Definition & The Two-Phase Lifecycle

LLM inference is the runtime evaluation process in which a trained language model processes prompt tokens and generates output text without adjusting network parameters. Inference divides into two distinct operational regimes:

• The Prefill Phase (Prompt Processing): The model ingests the entire input prompt of length \(T_{\text{in}}\) concurrently. All prompt tokens are processed through parallel matrix multiplications (GEMM). Because computation scales with the number of prompt tokens, prefill saturates GPU tensor cores and is primarily compute-bound.
• The Decode Phase (Token Generation): The model generates output tokens sequentially in an autoregressive loop. Generating each token requires loading the full model weight matrices from high-bandwidth memory (HBM) into on-chip cache to execute a single matrix-vector multiplication (GEMV). Because arithmetic work per weight load is minimal, decode is heavily memory-bandwidth bound.

2. Arithmetic Intensity & The Roofline Model

The operational limit of each phase is described by the Roofline Model, which relates arithmetic intensity (FLOPs per byte transferred from memory) to system performance bounds:

In single-sequence generation, a 70B parameter model in FP16 format occupies 140 GB of VRAM. Producing one token requires transferring all 140 GB across the memory bus. On an accelerator with 2.0 TB/s memory bandwidth, theoretical single-user decode speed cannot exceed \(2000 / 140 \approx 14.3\) tokens per second, regardless of compute FLOP head-room [1].

3. The Key-Value (KV) Cache

To prevent recalculating key and value projections for preceding tokens at every decode iteration, inference engines store calculated keys and values in GPU memory as the KV Cache.

The memory consumed by the KV cache per token across the network is:

$$\text{KV Bytes per Token} = 2 \times 2 \times n_{\text{layers}} \times n_{\text{heads}} \times d_k \times \text{precision bytes}$$

For large batch sizes and extended context windows, the KV cache footprint quickly surpasses the parameter footprint of the model itself. Several architectural advances compress this footprint:

• Multi-Query Attention (MQA): Shares a single key and value head across all query heads, reducing KV cache size by a factor of \(h\).
• Grouped-Query Attention (GQA): Partitions query heads into \(g\) groups, with each group sharing one key and value head (e.g., Llama 3).
• Multi-Head Latent Attention (MLA): Projects keys and values into low-rank compressed latent spaces, reducing KV memory footprint by up to 80% (DeepSeek-V3).

4. Serving Optimizations & Scheduling

Modern inference servers employ specialized scheduling and memory management frameworks:

• Continuous Batching: Traditional batching requires waiting for the slowest sequence in a batch to complete. Continuous (iteration-level) batching ejects completed sequences and injects new requests at each token generation step, eliminating idle padding waste (Yu et al., 2022) [2].
• PagedAttention: Inspired by virtual memory paging in operating systems, PagedAttention stores KV cache tensors in non-contiguous physical memory blocks rather than allocating rigid contiguous slabs. This reduces memory fragmentation from over 60% to under 4%, multiplying serving capacity (Kwon et al., 2023) [3].
• Speculative Decoding: A fast, lightweight draft model predicts candidate tokens in parallel, which the larger target model verifies in a single forward pass. Because verification is compute-bound rather than memory-bound, generation accelerates 2x to 3x without output distribution changes (Leviathan et al., 2023) [4].
• Chunked Prefill: Splits long input prompts into manageable chunks, interleaving them with decode iterations to prevent prefill bursts from stalling decode latency.

5. Serving Performance Metrics

Production inference performance is measured across three primary service-level indicators: