Systems Engineering · Machine Learning

Time to First Token

Reference entry · last updated August 26, 2026

Time to First Token (TTFT) is the latency duration between the moment a client submits a prompt to a language model service and the moment the client receives the first generated response token.[1] In generative AI systems, TTFT reflects the computational cost of prompt prefill and queuing delays, serving as the primary metric for perceived interactive responsiveness.[2]

The prefill and decode phase division

Large language model inference occurs in two computationally asymmetric stages:[1]

Prefill phase (prompt ingestion): The model ingests all \(N_{\text{in}}\) tokens of the input context in parallel, computing Key and Value tensors for every layer and writing them into the KV cache. Because all tokens are known upfront, this phase executes dense matrix multiplications (GEMM), achieving high arithmetic intensity on GPU tensor cores.
Decode phase (generation): The model generates completion tokens one by one autoregressively. Each new token requires evaluating the attention mechanism across all previous tokens, reading the entire model weights and KV cache from High Bandwidth Memory (HBM) for every single token step.[3]

TTFT measures the duration required to complete network ingress, queuing, and the entire prefill phase.

Time Per Output Token (TPOT)

While TTFT evaluates initial response delay, Time Per Output Token (TPOT) measures the generation speed of subsequent tokens:[2]

\[\text{TPOT} = \frac{\text{Total Latency} - \text{TTFT}}{N_{\text{out}} - 1}\]

where \(N_{\text{out}}\) is the count of generated output tokens. The inverse of TPOT represents generation throughput per user stream (in tokens per second). Total end-to-end latency \(T_{\text{total}}\) is expressed as:[2]

\[T_{\text{total}} = \text{TTFT} + (N_{\text{out}} - 1) \times \text{TPOT}\]

Hardware bottlenecks and arithmetic intensity

The operational constraints of TTFT and TPOT derive from the roofline model of computing:[4]

Prefill is compute-bound: High arithmetic intensity (FLOPs per byte of memory access). As prompt context scales, TTFT grows with prompt length \(O(N_{\text{in}})\) or \(O(N_{\text{in}}^2)\) in un-optimized attention implementations.
Decode is memory-bandwidth bound: Low arithmetic intensity. During decoding, GPU utilization is limited by the speed at which memory controllers transfer weight matrices from HBM to on-chip SRAM.

Optimization techniques

Systems employ distinct architectural patterns to minimize TTFT:[1]

Prompt caching: Storing pre-computed KV cache activations for static prefixes (such as system instructions, tool definitions, and few-shot examples). Reused prefixes bypass prefill computation, reducing TTFT by 50% to 90%.[5]
Chunked prefill: Splitting long prompt prefill tasks into smaller token chunks that are interleaved into decode batches, preventing head-of-line blocking for concurrent users.
Speculative decoding: Using a lightweight draft model to speculate multiple tokens in parallel, which a larger model verifies in a single forward pass, reducing effective TPOT.[6]

References

↑ Mitra, Sampriti. System Design for the LLM Era: Patterns and Principles for Production-Grade AI Architecture. Packt Publishing, 2026. Ch. 2: "Core Architectural Patterns for LLM System Design."
↑ Pope, Reiner, et al. "Efficiently Scaling Transformer Inference." Proceedings of Machine Learning and Systems (MLSys), vol. 5, 2023.
↑ Kwon, Woosuk, et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP, 2023.
↑ Williams, Samuel, Andrew Waterman, and David Patterson. "Roofline: An Insightful Visual Performance Model for Multicore Architectures." Communications of the ACM, vol. 52, no. 4, 2009, pp. 65–76.
↑ Anthropic. "Prompt Caching in Claude 3.5: Architecture and Latency Benchmarks," 2024.
↑ Leviathan, Yaniv, Matan Kalman, and Yossi Matias. "Fast Inference from Transformers via Speculative Decoding." ICML, 2023.