Machine Learning · Transformers

Transformer Block

Reference entry · last updated September 19, 2026

Transformer block (also called a Transformer layer) is one identical unit in the encoder or decoder stack. It maps a sequence of \(d_{\text{model}}\) vectors to another sequence of the same shape.[1] The full Transformer architecture is the tokenizer, the stack of these blocks, and the language modeling head.

1. First Principles: Shape-Preserving Residual Layer

A block takes a matrix \(X \in \mathbb{R}^{n \times d_{\text{model}}}\) and returns a matrix of the same size. Sequence length \(n\) and model width \(d_{\text{model}}\) do not change. Because the map preserves shape, \(N\) copies can be stacked without resizing the sequence.[1]

Each sub-layer is wrapped by a residual add. The residual keeps the identity path open, so the sub-layer computes a correction to \(x\) rather than a replacement for \(x\). In the 2017 paper the correction is then layer-normalized:

Vaswani et al. set \(d_{\text{model}} = 512\).[1]

2. Encoder and Decoder Sub-Layers

The encoder is a stack of \(N = 6\) identical layers. Each encoder layer has two sub-layers: multi-head self-attention, then a position-wise fully connected feed-forward network. A residual connection wraps each sub-layer, then layer normalization is applied.[1]

The decoder is also \(N = 6\) identical layers. It adds a third sub-layer: multi-head attention over the encoder stack (encoder-decoder attention). Decoder self-attention is masked so that position \(i\) cannot attend to subsequent positions.[1]

Decoder-only language models keep the masked self-attention and the feed-forward sub-layers and omit encoder-decoder attention. That configuration is described on the Transformer architecture page. This page stays on one layer.

3. Self-Attention

The first sub-layer is multi-head self-attention. Queries, keys, and values all come from the same sequence. The scaled dot-product formula, the multi-head split, and causal masking are defined on the self-attention page. This block page does not re-derive those projections.

In the encoder, every position may attend to every position. In the decoder, the mask zeros out future positions before the softmax. Encoder-decoder attention uses decoder queries against encoder keys and values.[1]

4. Position-Wise Feed-Forward Network

The second sub-layer (third in the decoder) is a fully connected network applied to each position separately and identically. Vaswani et al. give equation (2):[1]

The inner width is \(d_{\mathrm{ff}} = 2048\) in the 2017 base model, four times \(d_{\text{model}}\). The ReLU sits between two linear maps. Positions share the same weights. They do not share activations: each row of \(X\) is transformed on its own.[1]

5. Residual Connection and Layer Normalization

Around each sub-layer the paper uses a residual connection, then layer normalization. The output of each sub-layer is \(\mathrm{LayerNorm}(x + \mathrm{Sublayer}(x))\), where \(\mathrm{Sublayer}(x)\) is the function implemented by the attention or feed-forward module.[1]

The add requires the sub-layer output to have the same width as \(x\). Multi-head attention therefore projects concatenated heads back to \(d_{\text{model}}\). The feed-forward network expands to \(d_{\mathrm{ff}}\) and contracts back to \(d_{\text{model}}\).

6. Stacking Identical Layers

Encoder and decoder each stack \(N = 6\) copies of the same layer, with separate parameters per copy.[1] Because one block preserves \(n\) and \(d_{\text{model}}\), the next block can take its output as input without a reshape. Depth is a count of residual sub-layers, not a change of sequence geometry.

The stack is not the whole model. Token IDs become vectors before the first block. After the last block, a language modeling head maps the top hidden vector to vocabulary logits.

7. Feed-Forward Layers as Memories

Geva et al. (2021) treat the feed-forward layer as a key-value memory. Each key correlates with textual patterns in the training examples. Each value induces a distribution over the output vocabulary. Lower layers tend to capture shallow patterns. Upper layers tend to capture more semantic ones.[2]

Those experiments use the WikiText-103 transformer language model of Baevski and Auli (2019). The paper does not prove that every Transformer feed-forward layer is a factual store. The memory reading is an empirical account of that model, not a universal claim.[2]

8. Mixture of Experts as an FFN Replacement

A mixture-of-experts module can replace the dense position-wise feed-forward network. It does not replace the whole block. Self-attention, the residual add, and layer normalization remain. Sparse experts take the place of the shared \(W_1, W_2\) maps at that sub-layer.

9. Limits

• The block cannot change sequence length or model width. Padding, packing, and the context window are properties of the surrounding stack, not of one layer.
• Self-attention inside the block still costs \(O(n^2 \cdot d)\) time and memory per layer. That bound is derived on the self-attention page.
• Causal masking is a decoder self-attention constraint. It is not a property of the feed-forward sub-layer.
• Geva et al. describe feed-forward layers as memories on one WikiText-103 language model. That finding is not a proof for every model.[2]
• Mixture of experts changes the feed-forward sub-layer. It does not by itself change residual topology or attention.