Systems Engineering · Computer Science · Operations Research

Throughput

Reference entry · last updated August 26, 2026

Throughput is the rate at which a system processes, produces, or delivers discrete units of output over a given interval of time.[1] Across computer architecture, telecommunications, queuing networks, and industrial operations, throughput measures the productive capacity of a pipeline per unit of time, expressed mathematically as completed units divided by total observation time.[2]

First principles and definitions

At its core, throughput \(X\) measures flow rate through a bounded system over time duration \(T\):

\[X = \frac{C}{T}\]

where \(C\) represents the count of successfully completed work units. The specific unit of measurement varies by domain:[1]

Telecommunications: Bits per second (bps) or packets per second (pps).
Transactional databases: Transactions per second (TPS) or queries per second (QPS).
Processor microarchitecture: Instructions per cycle (IPC) or floating-point operations per second (FLOPS).
Large language model serving: Tokens per second (TPS) or requests per minute (RPM).
Manufacturing and software delivery: Units shipped per day or pull requests merged per week.

Throughput is distinct from two related concepts:[2]

Latency (or response time): The elapsed time required for a single unit of work to traverse the system from arrival to completion.
Bandwidth (or maximum capacity): The theoretical upper limit on throughput that a system channel can sustain under ideal, contention-free conditions. Bandwidth represents potential capacity; throughput represents realized delivery.

Little's Law and queuing dynamics

In queuing theory, John D. C. Little proved the foundational mathematical identity governing all stable, conservative systems in equilibrium:[3]

\[L = \lambda W\]

where:

\(L\) is the average number of items inside the system concurrently (concurrency, inventory, or work-in-progress).
\(\lambda\) is the long-term average arrival and departure rate (throughput).
\(W\) is the average time an item spends inside the system (mean residence time or total latency).

Little's Law establishes that throughput is strictly constrained by concurrency and latency:

\[\lambda = \frac{L}{W}\]

To increase throughput, a system must either increase concurrency \(L\) (processing more items in parallel) or decrease residence time \(W\) (processing each item faster). If \(W\) increases due to internal queuing delays while \(L\) remains fixed, throughput \(\lambda\) must drop.[3]

The throughput-latency relationship

The relationship between offered load, throughput, and latency exhibits three distinct operational regimes:[4]

Linear regime (low load): When arrival rate is low, resources remain underutilized. Throughput scales linearly with offered load, and latency remains near the baseline processing time (\(W \approx W_{\text{service}}\)).
Knee of the curve (optimal utilization): As load reaches resource saturation (\(N^*\)), throughput reaches its maximum ceiling (\(X_{\max}\)). Hardware utilization approaches 100%, and processing pipelines run at peak efficiency without accumulating significant queues.[4]
Queuing saturation regime (overload): Adding load beyond the knee creates queue backlogs. Throughput stays flat or degrades due to context-switching overhead, while latency increases non-linearly. According to Kingman's formula for single-server queues, wait time diverges toward infinity as utilization \(\rho \to 1\):[5]

\[W_q \approx \left(\frac{\rho}{1 - \rho}\right) \left(\frac{C_a^2 + C_s^2}{2}\right) \tau\]

Concurrency limits and scaling laws

Adding parallel processing units does not scale throughput indefinitely due to communication overhead and serial bottlenecks.[6]

Amdahl's Law defines the maximum speedup \(S(N)\) achievable by adding \(N\) parallel processors when a fraction \(s\) of the work must execute serially:[6]

\[S(N) = \frac{1}{s + \frac{1 - s}{N}}\]

As \(N \to \infty\), maximum throughput is bounded strictly by \(\frac{1}{s}\). If 5% of a workload is strictly serial (\(s = 0.05\)), the maximum possible speedup is 20x, regardless of how many parallel cores or agent workers are provisioned.[6]

Gunther's Universal Scalability Law (USL) extends Amdahl's Law by accounting for inter-worker coherence penalties (such as lock contention and cache invalidation protocols):[7]

\[C(N) = \frac{N}{1 + \alpha(N - 1) + \beta N(N - 1)}\]

where \(\alpha\) represents contention (serialization) and \(\beta\) represents cross-node coherence overhead. When \(\beta > 0\), scaling past an optimal node count causes throughput to decline (retrograde scalability).[7]

Throughput in computer architecture and AI inference

Modern machine learning serving systems illustrate the architectural mechanics of throughput:[8]

Large language model inference is divided into two distinct computational phases with opposing bottlenecks:[9]

Prefill phase (prompt ingestion): Processes all input tokens in parallel. This phase is compute-bound, maximizing Matrix Multiplication (GEMM) utilization on GPU Tensor Cores to deliver high Time-to-First-Token (TTFT) throughput.
Decode phase (autoregressive token generation): Generates one token at a time sequentially per active request. This phase is memory-bandwidth bound, reading entire multi-gigabyte weight tensors from High Bandwidth Memory (HBM) into SRAM for every single generated token.[9]

To maximize total serving throughput, modern runtimes employ specific architectural optimizations:[8]

Continuous / iteration-level batching: Rather than waiting for an entire batch to complete, requests are dynamically injected and retired at every token iteration step.
PagedAttention: Partitions the Key-Value (KV) cache into non-contiguous virtual memory blocks, eliminating internal fragmentation and increasing concurrent batch capacity by 2x to 4x.[8]
Chunked prefill: Interleaves long prompt prefill chunks into decode batches to prevent single long prompts from stalling streaming decode throughput for concurrent users.

Theory of Constraints and workflow throughput

In operations research, Eliyahu M. Goldratt formalized throughput in organizational systems via the Theory of Constraints:[10]

Every system contains exactly one primary constraint (bottleneck) that limits overall throughput.
Total system throughput is strictly determined by the capacity of the bottleneck stage.
Any local optimization made at a non-bottleneck stage produces excess inventory (WIP) without increasing finished throughput.[10]

In software engineering pipelines, speeding up code generation (via AI coding agents) without expanding human review or automated testing capacity shifts the bottleneck downstream, creating review backlogs without increasing deployed production throughput.[11]

Measurement pitfalls and honest capacity planning

Evaluating system capacity requires avoiding common measurement errors:[4]

Single-user microbenchmarking: Measuring single-stream performance on an idle system reports minimum latency, not maximum throughput. Real throughput must be evaluated across concurrent load sweeps.
Ignoring operation quality: Comparing raw operation counts without accounting for error rates, retries, or code churn produces false throughput signals. High-throughput generation that fails downstream verification decreases net effective throughput.
Ignoring tail latency: Sizing systems to average throughput leaves them vulnerable to queue collapse during traffic bursts. Capacity planning must evaluate throughput under 95th and 99th percentile load distributions.[4]

References

↑ Patterson, David A., and John L. Hennessy. Computer Organization and Design: The Hardware/Software Interface. 6th ed., Morgan Kaufmann, 2020. Ch. 1: "Computer Abstractions and Technology."
↑ Kurose, James F., and Keith W. Ross. Computer Networking: A Top-Down Approach. 8th ed., Pearson, 2021. Ch. 1: "Throughput in Computer Networks."
↑ Little, John D. C. "A Proof for the Queuing Formula: \(L = \lambda W\)." Operations Research, vol. 9, no. 3, 1961, pp. 383–387. https://doi.org/10.1287/opre.9.3.383
↑ Gregg, Brendan. Systems Performance: Enterprise and the Cloud. 2nd ed., Addison-Wesley, 2020. Ch. 2: "Methodologies and Concepts."
↑ Kingman, J. F. C. "The Single Server Queue in Heavy Traffic." Mathematical Proceedings of the Cambridge Philosophical Society, vol. 57, no. 4, 1961, pp. 902–904.
↑ Amdahl, Gene M. "Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities." AFIPS Conference Proceedings, vol. 30, 1967, pp. 483–485. https://doi.org/10.1145/1465482.1465560
↑ Gunther, Neil J. Guerrilla Capacity Planning: A Tactical Approach to Planning for Highly Scalable Applications and Services. Springer, 2007.
↑ Kwon, Woosuk, et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), 2023. https://doi.org/10.1145/3600006.3613165
↑ Mitra, Sampriti. System Design for the LLM Era: Patterns and Principles for Production-Grade AI Architecture. Packt Publishing, 2026. Ch. 2: "Core Architectural Patterns for LLM System Design."
↑ Goldratt, Eliyahu M., and Jeff Cox. The Goal: A Process of Ongoing Improvement. North River Press, 1984.
↑ Faros AI. "The 2026 Engineering Productivity and AI Acceleration Report," 2026.