Latency (Systems and Computing)

Reference entry · last updated September 7, 2026

Latency is the time interval elapsed between the initiation of an operation or request and the observation of its initial result or complete delivery. While throughput measures the aggregate rate of completed production units over time, latency quantifies individual turnaround duration or delay. In computer systems, network engineering, distributed services, and machine learning inference, latency is decomposed into physical propagation, transmission delay, serialization, processing execution time, and queuing overhead.

1. First Principles: Latency vs. Throughput

In computer architecture, David A. Patterson and John L. Hennessy distinguish between performance metrics: response time (latency) is the total elapsed time required to complete a single task, whereas throughput is the total amount of work accomplished per unit time [1].

The relationship is often illustrated by physical pipeline dynamics:

• Latency: The travel time required for a single data unit to traverse a channel from source to destination (measured in seconds, milliseconds, or nanoseconds).
• Throughput: The volumetric rate at which data units exit the channel per unit time (measured in operations per second, bytes per second, or tokens per second).

A high-throughput system is not necessarily low-latency. For instance, transporting terabytes of data across continents via physical magnetic tape shipments delivers massive throughput, but exhibits extreme latency (measured in hours or days). Conversely, an audio streaming connection requires minimal latency (sub-10ms) while demanding modest throughput.

2. Structural Components of Latency

In network and distributed systems engineering, end-to-end latency \(L_{\text{total}}\) decomposes into four additive components [2]:

2.1 Propagation and Transmission Delays

• Propagation Delay (\(T_{\text{prop}}\)): The physical duration required for an electromagnetic signal to travel through a physical medium (copper cable, optical fiber, or free space). Bounded fundamentally by the speed of light in vacuum (\(c \approx 3 \times 10^8\,\text{m/s}\)) and the refractive index of glass (\(n \approx 1.47\)), light travels through fiber optic strands at approximately \(2 \times 10^8\,\text{m/s}\) (\(\approx 5\,\mu\text{s}\) per kilometer).
• Transmission Delay (\(T_{\text{trans}}\)): The time required for a network interface card to push all packet bits onto the wire. Determined strictly by packet size \(B\) and link bandwidth \(R\): \[T_{\text{trans}} = \frac{B}{R}\]

2.2 Processing and Memory Delays

• Processing Delay (\(T_{\text{proc}}\)): The CPU time required to inspect headers, check error-correcting codes, execute business logic, or deserialize data buffers.
• Memory Hierarchy Access: Instruction execution is bounded by cache access latency. While L1 cache lookups complete in \(\approx 1\,\text{ns}\), main memory (DRAM) accesses require \(50\text{--}100\,\text{ns}\), and disk/network reads require milliseconds.

2.3 Queuing Delays

Queuing Delay (\(T_{\text{queue}}\)): The time a job spends waiting in buffers or scheduler queues before an idle execution worker begins servicing it. While propagation, transmission, and processing times are relatively deterministic for a given payload size, queuing delay is highly variable and depends on resource contention and arrival distribution.

3. Mathematical Models and Queuing Dynamics

Queuing theory establishes formal mathematical bounds governing how latency escalates under increasing operational load.

3.1 Little's Law and Capacity Bounds

John D. C. Little proved in 1961 that for any stable queuing system, the long-term average number of concurrent items inside the system \(L\) equals the arrival rate (throughput) \(\lambda\) multiplied by the average time \(W\) an item spends in the system [3]:

Little's Law demonstrates that average latency \(W\) cannot be reduced without either reducing concurrency \(L\) (limiting in-flight work) or increasing processing rate \(\lambda\).

3.2 Kingman's Heavy Traffic Approximation

In single-server queuing systems (\(G/G/1\)), J. F. C. Kingman (1961) proved that average waiting time \(W_q\) diverges non-linearly as system utilization \(\rho\) approaches 100% (\(\rho \to 1\)) [4]:

where \(\rho = \lambda \tau\) is utilization, \(\tau\) is mean service time, \(c_a\) is the coefficient of variation of arrivals, and \(c_s\) is the coefficient of variation of service times.

As utilization climbs past 80%, the term \(\frac{\rho}{1 - \rho}\) escalates rapidly. Running hardware or agent pipelines near 100% capacity guarantees queuing collapse and severe latency spikes.

4. Tail Latency and Percentile Distributions

In production environments, mean (average) latency obscures user experience because distribution tails follow heavy-tailed or multi-modal patterns. Systems performance engineers evaluate latency via percentiles:

• p50 (Median): The latency experienced by the 50th percentile of requests.
• p95 / p99: The latency threshold that 95% or 99% of requests beat, exposing queue contention and garbage collection pauses.
• p99.9 / Maximum: The extreme tail, driven by server timeouts, thread pool starvation, or packet retransmission.

4.1 The Amplification of Tail Latency at Scale

In modern microservice and distributed retrieval architectures, a single user request fans out to \(M\) distinct backend workers (such as database shards or vector search partitions). In their landmark paper The Tail at Scale (2013), Jeffrey Dean and Luiz André Barroso demonstrated that overall user latency is determined by the slowest responding shard [5]:

When a request queries 1,000 servers in parallel, virtually 100% of user interactions will experience the slowest server's 99th percentile delay.

4.2 Tail-Tolerant Architectural Techniques

Dean and Barroso outlined techniques to tame tail latency in distributed architectures:

• Hedged Requests: The client dispatches a request to one replica; if no response is received by the 95th percentile expected latency, a duplicate request is sent to a second replica. The client uses whichever response arrives first and cancels the other.
• Tied Requests: The client enqueues identical requests on two replicas simultaneously. When one server begins execution, it notifies the second server to cancel its duplicate work.
• Micro-Partitioning: Dividing work into granular task chunks to avoid head-of-line blocking on uneven workloads.

5. Latency in Machine Learning and LLM Serving

In large language model serving, latency divides into two distinct operational phases [7]:

• Time to First Token (TTFT): The latency required to ingest the input prompt and run the initial matrix multiplications (the prefill phase). Bound primarily by compute and memory bandwidth across total prompt length.
• Inter-Token Latency (ITL) / Time Per Output Token (TPOT): The elapsed time to emit each subsequent token (the decode phase). Because autoregressive decoding emits one token at a time, each step requires reloading model weights from High Bandwidth Memory (HBM), making ITL memory-bandwidth bound.

Techniques like continuous batching, chunked prefill, and speculative decoding specifically balance the trade-off between maximizing server throughput and minimizing per-user generation latency.