← Reference · Nestor G Pestelos Jr · Print this page

Systems Engineering · Machine Learning

Time Per Output Token

Reference entry · last updated September 8, 2026

Time Per Output Token (TPOT), also measured as Inter-Token Latency (ITL), is the time elapsed between emitting successive tokens during the autoregressive decode phase of a large language model. While Time to First Token (TTFT) dictates initial interactive response delay, TPOT governs generation velocity and human-perceived streaming smoothness across multi-token responses.

1. First Principles: Memory-Bandwidth Bottlenecks in Autoregressive Decode

Autoregressive transformer inference proceeds in two fundamentally distinct execution regimes: prefill and decode [1]. The prefill phase evaluates prompt tokens in parallel, achieving high arithmetic intensity (floating-point operations performed per byte of data loaded from memory). In contrast, the decode phase is strictly memory-bandwidth bound.

During decoding, each forward pass generates exactly one new token per sequence. To emit that token, the hardware execution units must stream the entire model parameter weight matrix \(P\) from High Bandwidth Memory (HBM) into on-chip static RAM (SRAM) and register files [2]. For a model with \(P\) parameters stored in 16-bit precision (2 bytes per parameter), loading weights requires reading \(2P\) bytes. The matrix-vector arithmetic performs approximately \(2P\) floating-point operations (FLOPs). The arithmetic intensity \(I_{\text{decode}}\) for batch size \(B = 1\) is therefore:

Modern accelerator hardware (such as an NVIDIA H100 GPU) provides approximately 3.35 TB/sec of memory bandwidth and up to 1,979 TFLOPs of FP16 tensor core compute. A processor capable of nearly 2,000 trillion operations per second but limited to 3.35 trillion bytes per second has a hardware balance point above 150 FLOPs/byte [3]. At 1 FLOP/byte, the compute units spend over 98% of their cycles idle, waiting for memory controllers to stream weights. TPOT is therefore determined almost entirely by parameter count and memory bandwidth rather than peak arithmetic capability.

2. Formulations and Measurement

2.1 TPOT vs. Inter-Token Latency

In systems benchmarks, Time Per Output Token (TPOT) is defined as the mean duration spent per output token over the entire generation sequence:

where \(T_{\text{total}}\) is total elapsed request latency, \(\text{TTFT}\) is time to first token, and \(N_{\text{out}}\) is the total count of emitted tokens. TPOT is often inverted to describe single-stream generation throughput:

While TPOT measures sequence-wide averages, Inter-Token Latency (ITL) evaluates instantaneous delay between consecutive individual tokens \(t_k\) and \(t_{k-1}\):

Per-token ITL exposes latency jitter caused by iteration-level scheduling, memory paging, and prefill interference. High-percentile ITL (P90, P99) reflects pauses or hitching that degrade human reading experience, even when average TPOT appears satisfactory.

2.2 End-to-End Latency Composition

Total request response duration decomposes cleanly into prefill latency and cumulative decode latency:

For conversational workloads where output lengths reach hundreds of tokens, cumulative decode time dominates end-to-end response latency.

3. The Batch Size and Throughput Trade-off

Serving systems batch multiple concurrent client requests to overcome memory-bandwidth starvation [1]. When batch size increases from 1 to \(B\), the accelerator reads model weights once from HBM and reuses them across all \(B\) tokens in parallel. Arithmetic intensity scales linearly with batch size:

where \(S_{\text{KV}}\) represents per-token Key-Value (KV) cache memory. Larger batch sizes increase total cluster throughput (tokens served per second per dollar). However, larger batches increase cache memory contention and lengthen each iteration step, gradually degrading per-user TPOT. Serving architectures operate on an efficient frontier balancing hardware cost against per-stream generation latency.

4. Engineering Strategies to Reduce TPOT

Engineers employ several hardware and algorithmic techniques to lower TPOT and improve generation responsiveness:

• Speculative Decoding: A small, fast draft model guesses candidate tokens, which the larger target model verifies in a single forward pass. Because verification parallelizes across speculative tokens, effective TPOT drops without changing output quality [4].
• Weight Quantization: Compressing weights from FP16 to INT8, INT4, or FP4 cuts byte transfer volume by 50% to 75%, directly reducing weight loading duration per decode iteration.
• Chunked Prefill: Slicing compute-dense prompt evaluations into smaller pieces prevents incoming queries from interrupting decode iterations, eliminating P99 ITL spikes.
• Disaggregated Serving: Physically separating prefill nodes from decode nodes (such as Splitwise and DistServe) isolates memory-bandwidth-bound decode clusters from compute-bound prefill workloads [5].