← Reference · Nestor G Pestelos Jr

Systems Architecture · Machine Learning Serving

Continuous Batching

Reference entry · last updated August 26, 2026

Continuous batching (also known as iteration-level scheduling) is an inference execution strategy in which requests are dynamically admitted to and retired from active compute batches at the granularity of individual token generation steps.[1] Unlike static batching, which locks processing slots until the longest request in the batch terminates, continuous batching reclaims GPU memory and hardware capacity immediately as individual requests complete.[2]

The static batching bottleneck

Traditional deep learning inference groups requests into uniform batches. In autoregressive generation, request sequence lengths vary widely: a short question might complete in 20 tokens, while a code synthesis query requires 1,000 tokens.[1]

Static batching incurs two structural inefficiencies:[2]

• Head-of-line blocking: Fast requests finish early but cannot return until the slowest request in the batch completes.
• Padding token waste: Shorter sequences must be padded with dummy tokens to match the longest sequence in the batch tensor, wasting GPU compute cycles on non-informative matrix multiplications.

Iteration-level scheduling mechanics

Continuous batching restructures the execution loop from request-level scheduling to iteration-level scheduling:[1]

1. At each forward pass iteration step \(t\), the inference engine evaluates all active sequences in the batch to generate one output token per sequence.
2. If a sequence outputs an end-of-sequence token (<|endoftext|>) or reaches its token limit, its execution slot and Key-Value (KV) cache memory are released immediately.
3. A new queued request is admitted into the freed slot at step \(t+1\), executing its prefill phase while ongoing requests continue their decode phase.[1]

PagedAttention and memory management

Continuous batching requires dynamic, non-contiguous memory management. Because sequence lengths are unpredictable, reserving maximum-context contiguous memory blocks on GPU VRAM wastes up to 60% to 80% of memory on unallocated reservation space (internal fragmentation).[2]

The PagedAttention algorithm adapts virtual memory paging to transformer KV caches:[2]

• KV cache activations are partitioned into fixed-size physical blocks (e.g. 16 tokens per block).
• A page table maps logical token positions to physical blocks located non-contiguously in VRAM.
• Physical blocks are allocated strictly on demand as new tokens are generated.

Throughput and latency improvements

By pairing continuous batching with virtual memory paging, production engines (such as vLLM and TensorRT-LLM) achieve 2x to 4x higher serving throughput (tokens per second) compared to static baseline systems under identical GPU hardware constraints.[2]

See also

• Throughput
• Time to First Token
• Tokens and Tokenization
• LLM Gateway

References

1. ↑ Yu, Gyeong-In, et al. "Orca: A Distributed Serving System for Transformer-Based Generative Models." Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2022.
2. ↑ Kwon, Woosuk, et al. "Efficient Memory Management for Large Language Model Serving with PagedAttention." Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP), 2023.
3. ↑ Mitra, Sampriti. System Design for the LLM Era: Patterns and Principles for Production-Grade AI Architecture. Packt Publishing, 2026. Ch. 2: "Core Architectural Patterns for LLM System Design."