Machine Learning · LLM Serving

Decode (LLM Inference)

Reference entry · last updated September 11, 2026

Decode (also called generation) is the second phase of transformer language-model inference: after prefill, the model emits output tokens one at a time. Each step conditions on all tokens so far and appends a new key/value pair to the KV cache.[1] Mean time per later token is time per output token (TPOT), also called inter-token latency.[2]

1. First Principles: Sequential Next-Token Steps

Autoregressive generation cannot evaluate future output positions in parallel: token \(t+1\) depends on the sample chosen at \(t\). Pope et al. call the loop decode or generation: \(L_{\mathrm{gen}}\) sequential forward passes, one new token per sequence in the batch per step.[1]

Each step still loads the model weights from HBM. With batch size 1 the matmul is essentially a matrix-vector product (GEMV). Arithmetic intensity is low, so decode is typically memory-bandwidth-bound: tokens per second cannot exceed roughly memory bandwidth divided by bytes of weights (and KV) that must be read each step.[1][2] DistServe states the same split: prefill processes many tokens in one step and tends to be compute-bound; each decode step processes one new token with similar I/O and is bandwidth-bound.[2]

Attention at step \(t\) reads cached keys and values for positions \(1 \ldots t-1\) instead of recomputing the prompt. Cache size therefore grows with generated length (and with the prompt, which was written in prefill). Two knobs cut that growth. MQA, GQA, and MLA shrink the KV stored per retained token. Sliding-window attention with a rolling buffer keeps only the last \(W\) positions, so cache size stops growing at the window; the per-token KV width does not shrink. See KV Cache and Sliding-Window Attention. Neither knob removes the need to load weights every step.

2. Metrics and Bottlenecks

End-to-end latency is TTFT plus TPOT times the number of generated tokens after the first.[2] Interactive chat cares about TPOT until it is faster than reading speed; summarization cares about TPOT for the whole document.[2]

Raising batch size amortizes weight loads across more tokens per step and can push decode toward compute-bound. It also multiplies KV memory. PagedAttention stores KV in non-contiguous blocks to cut reservation waste so larger batches fit.[4]

Pope et al. report 29 ms per generated token on PaLM 540B (int8 weights, 64 TPU v4 chips, long 2048-token context) as a point on their Pareto curve. That is a measured configuration, not a decode constant.[1]

3. Batching and Disaggregation

Iteration-level (continuous) batching admits new sequences at token granularity so finished requests leave the batch without waiting for the longest output. Mixing a full prefill into that batch still lengthens every decode step in the mix. Chunked prefill and prefill/decode disaggregation are the usual mitigations; see Prefill and Chunked Prefill.[2][3]

4. Other Uses of Decode

In coding, decode often means turning a bitstream or ciphertext back into data. This entry is only the LLM generation phase. It is not the decoder stack of an encoder-decoder transformer by itself, though that stack also runs token-by-token at inference.