Machine Learning · LLM Serving

Prefill (LLM Inference)

Reference entry · last updated September 11, 2026

Prefill is the first phase of transformer language-model inference: the model processes every token of the prompt in one parallel forward pass and writes the resulting key and value tensors into the KV cache.[1] The first generated token is produced at the end of this pass. Wall-clock time from request arrival to that token is time to first token (TTFT).[2]

1. First Principles: Parallel Prompt Evaluation

A decoder-only transformer maps a sequence of tokens to next-token distributions. Training and prefill can evaluate many positions at once because the prompt is known. Pope et al. call this step prefill: a single forward pass over all \(B \times L_{\mathrm{input}}\) prompt tokens, as distinct from the sequential loop that later emits output tokens.[1]

That pass is a stack of dense matrix multiplies (GEMM) over the full prompt length. Arithmetic intensity (FLOPs per byte loaded from HBM) is high when \(L_{\mathrm{input}}\) is large, so the phase is typically compute-bound on GPU tensor cores rather than memory-bandwidth-bound.[1][2] DistServe notes that for a 13B model, prefilling a 512-token sequence already saturates an NVIDIA A100; extra tokens in the same step add superlinear work while still moving similar weight I/O.[2]

Outputs that later decode needs are stored as KV activations per layer and per prompt token. Recomputing them every generation step would repeat the prompt matmuls. Caching them is what makes decode a smaller GEMV plus attention over stored keys and values. See Decode (LLM Inference).

2. Metrics and Bottlenecks

TTFT is the duration of prefill plus queueing, not including later tokens.[2] It grows with prompt length, model size, and contention from other jobs on the same GPU. Pope et al. report prefill and generate separately because the two phases have different partitioning optima: large-token batches in prefill can use weight-gathered layouts and high model FLOPS utilization, while generation keeps a small token count per step.[1]

Those layout numbers (for example 76% MFU on PaLM 540B prefill in their TPU v4 study) are hardware- and model-specific. They are not a universal prefill efficiency.

3. Scheduling with Decode

Online servers often colocate prefill and decode on the same accelerators and mix them with continuous batching. A long prefill in that mix delays in-flight decode steps (prefill-decode interference).[2]

Two common responses:

• Chunked prefill splits a long prompt into token chunks and piggybacks decode tokens in the same iteration, trading some extra KV loads for smoother inter-token latency.[3]
• Disaggregation runs prefill and decode on different GPUs and ships KV state between them, so each phase can use its own batch size and parallelism. DistServe reports higher per-GPU goodput under joint TTFT and TPOT SLOs in that design; the gain is an evaluation result, not a law of prefill.[2]

4. Other Uses of Prefill

In web forms, prefill means inserting saved field values. This entry is only the LLM serving phase. It is not pre-training.