← Reference · Nestor G Pestelos Jr · Print this page

Reference Document

Transformer Architecture

A citable reference on the Transformer architecture: self-attention formulations, multi-head projection layers, residual normalization, and sequence representations.

1. Definition & Motivation

The Transformer architecture is a deep neural network design introduced by Vaswani et al. (2017) that models sequence dependencies entirely through self-attention operations and feed-forward projections, omitting recurrent loops and convolutional filters [1].

Prior sequence models, such as Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs), processed tokens sequentially. That serial bottleneck prevented full hardware parallelization across training sequences. Information had to pass through sequential hidden states \(h_t\), leading to vanishing gradients across long contexts. The Transformer processes all sequence positions concurrently in training, reducing the path length for distant token interactions to \(O(1)\) operations.

2. Scaled Dot-Product Attention

The primary primitive of the Transformer is scaled dot-product attention. Given input representations packed into Query matrix \(Q\), Key matrix \(K\), and Value matrix \(V\), attention computes weighted averages of values based on query-key inner products:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Here \(d_k\) denotes the dimension of the keys. The scaling factor \(\frac{1}{\sqrt{d_k}}\) counteracts growth in the dot-product magnitude as dimension size expands. Without this scaling, large dot products push the softmax function into regions with tiny gradients, stalling training progress.

In causal or autoregressive decoders, an attention mask with \(-\infty\) entries is added to upper-triangular logits before the softmax step. This ensures position \(i\) cannot attend to future tokens \(j > i\).

3. Multi-Head Attention

Rather than computing a single attention distribution with dimension \(d_{\text{model}}\), multi-head attention projects queries, keys, and values \(h\) times using learned parameter matrices:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O$$

$$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$$

Projections map inputs into lower-dimensional subspaces (\(d_k = d_v = d_{\text{model}} / h\)). Multiple heads allow the model to attend simultaneously to information from different representation subspaces at different positions, capturing syntax, co-reference, and factual relations concurrently.

4. Feed-Forward Networks & Normalization

Each attention block is paired with a position-wise feed-forward network (FFN). The FFN consists of two linear transformations separated by a non-linear activation (historically ReLU, modernized to GELU or SwiGLU):

$$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$$

Every sub-layer (attention and FFN) incorporates a residual skip connection followed by layer normalization:

$$\text{Output} = \text{LayerNorm}(x + \text{SubLayer}(x))$$

Modern implementations often adopt Pre-LayerNorm (normalizing inputs before sub-layers) or Root Mean Square Normalization (RMSNorm) for superior numerical stability during large-scale pre-training.

5. Positional Representation

Because attention operates as a permutation-equivariant set operation, the architecture possesses no built-in spatial order. Positional information must be explicitly injected into input token embeddings.

• Sinusoidal Encodings: Fixed trigonometric wave functions across dimension channels (Vaswani et al., 2017).
• Learned Absolute Embeddings: Dedicated position vectors optimized during training (Radford et al., 2018).
• Rotary Position Embedding (RoPE): Multiplicative rotation of query and key representations in complex space, encoding relative distance through inner products (Su et al., 2024) [2]. RoPE dominates contemporary frontier decoders.

6. Architectural Configurations