Machine Learning · Transformers

Sliding-Window Attention (Transformers)

Reference entry · last updated September 11, 2026

Sliding-window attention (SWA) is a sparse self-attention pattern in which each position attends only to a fixed-size neighborhood of nearby tokens, rather than to the full sequence.[1] With window width \(w\), time and memory scale as \(O(n w)\) in sequence length \(n\), instead of the \(O(n^2)\) cost of dense transformer attention.[2]

1. First Principles: Local Receptive Fields

In the original transformer, scaled dot-product attention at every position mixes queries with keys and values from all positions (with a causal mask in autoregressive models).[2] The \(QK^\top\) product is \(n \times n\). That quadratic term bounds context length on a given accelerator.

A sliding window replaces the full row of the attention matrix with a band of width \(w\). Longformer states the local rule as: given window size \(w\), each token attends to \(\frac{1}{2}w\) tokens on each side, with complexity \(O(n \times w)\).[1] Causal SWA (as in Mistral 7B) restricts the neighborhood to the previous \(W\) tokens of the prior layer.[3]

A single window cannot see the whole document. Stacked layers expand the receptive field. Longformer: with \(\ell\) layers and fixed \(w\), the top layer’s field is \(\ell \times w\).[1] Mistral: hidden state \(h_i\) at layer \(k\) attends positions \([i-W, i]\) of the previous layer, so input-layer tokens up to distance \(W \times k\) can influence the last layer. For \(W = 4096\) they give a theoretical span of about 131K tokens.[3]

That stacked-field argument is architectural, not a guarantee that every long-range dependency is used. Information has to survive the intervening layers. Models that need a privileged global token still add a separate global pattern (below).

2. Window, Dilation, and Global Tokens

Longformer treats SWA as a drop-in replacement for dense self-attention and combines three patterns:[1]

• Sliding window. Local band around each token (Fig. 2b in the paper).
• Dilated sliding window. Gaps of dilation \(d\) inside the window, analogous to dilated convolutions, expanding the field to \(\ell \times d \times w\) without raising \(w\).
• Global + window. A small set of positions (for example [CLS], or all question tokens in QA) attend to the full sequence, and every token attends to those positions. Count of global tokens is independent of \(n\), so the mix remains linear in \(n\).

Sparse Transformer used related local and strided block patterns for long sequences, mainly on autoregressive language modeling.[4] Longformer cites that line and adds task-chosen global tokens plus continued pretraining from RoBERTa for document NLP.[1] Mistral 7B cites both Sparse Transformer and Longformer for SWA, and pairs SWA with grouped-query attention.[3]

Longformer also uses separate \(Q,K,V\) projections for the window scores and the global scores. Their WikiHop ablations treat those extra projections as necessary for the reported transfer-learning numbers, not as optional polish.[1]

3. Inference and KV Cache

Dense decode stores a key/value vector per layer per past token, so cache memory grows with context. SWA bounds the live neighborhood. Mistral implements a rolling buffer cache of size \(W\): keys and values at timestep \(i\) overwrite slot \(i \bmod W\). After \(i > W\), cache size stops growing. They report an 8× cache reduction on 32k-token sequences at \(W = 4096\) in that setup, without a measured quality drop in the paper’s statement of the method.[3]

Prefill of a long prompt can be chunked at window size. Each chunk attends to itself (causal) and to the rolling cache, and does not attend tokens already outside the window.[3]

This cache bound is local KV only. A model that also keeps a global branch (indexer keys, compressed global KV) still pays that global storage. See KV Cache.

4. Hybrids with Global Sparse Retrieval

Serving systems that keep million-token agent prefixes often pair SWA with a second, compressed global path so local mixing stays cheap while some tokens remain addressable outside the window. DeepSeek-V4.1-Flash uses SWA in every layer. The first two layers are SWA-only; remaining layers add a global branch (main KV and indexer K). They state that for a fixed window, SWA KV storage does not grow with sequence length, so global KV dominates HBM at long length.[5]

V4.1-Flash further describes SWA Bounded Replay: approximate reconstruction of per-layer SWA state by replaying about \(n_{\mathrm{win}}\) recent tokens rather than \(L \times n_{\mathrm{win}}\), so SWA KV need not be persisted to SSD. The report claims negligible quality drop; that measurement is not reproduced here.[5]

Those ratios (runtime KV about 1/4 of V4-Flash, persistent KV about 1/8) are developer-reported for that checkpoint. They are not properties of SWA in general.

5. Other Uses of SWA

Outside transformers, SWA is a common acronym for sliding-window average, software architecture, and other terms. This entry is only the attention pattern. It is not neurological attention (Attention) and not a full substitute for dense attention on short sequences where \(n\) already fits in HBM.