Nestor G Pestelos Jr · Reference · Print this page

Embeddings

Published September 3, 2026 · Machine Learning & Representation Theory

Vector embeddings are dense numerical representations of discrete entities (such as tokens, sentences, code, or images) in a continuous high-dimensional vector space. By projecting discrete concepts into continuous geometries, embeddings enable machine learning models to quantify semantic similarity, capture associative analogies, and conduct sub-linear nearest neighbor retrieval across massive corpora.

1. Theoretical Motivation

In classical computational linguistics, words were treated as atomic symbols represented by orthogonal one-hot vectors [1]. In a vocabulary of size \(|V|\), each word \(w_i\) is represented by a vector \(\mathbf{x}_i \in \{0, 1\}^{|V|}\) where exactly one entry is 1 and all others are 0. This formulation exhibits two fundamental flaws:

• Dimensional curse: The dimensionality grows linearly with vocabulary scale, requiring tens or hundreds of thousands of sparse dimensions.
• Orthogonal isolation: Every one-hot vector is completely orthogonal to every other: \(\mathbf{x}_i \cdot \mathbf{x}_j = 0\) for all \(i \neq j\). Consequently, the representation cannot capture that "physician" and "doctor" share semantic meaning, while "physician" and "submarine" do not.

Modern representation learning addresses this limitation through the distributional hypothesis, formulated by linguist J.R. Firth in 1957: "You shall know a word by the company it keeps" [2]. Dense vector embeddings map discrete vocabulary items into a low-dimensional continuous manifold \(\mathbb{R}^d\) (where typically \(d \in [256, 4096]\)), such that words appearing in similar contextual distributions are positioned close to one another in latent geometric space.

2. Mathematical Formulation & Geometry

Given an input token index \(i \in \{1, \dots, |V|\}\), the embedding operation corresponds to a matrix lookup in an embedding weight matrix \(\mathbf{W}_E \in \mathbb{R}^{|V| \times d}\):

where \(\mathbf{x}_i\) is the one-hot indicator vector for token \(i\), and \(\mathbf{e}_i \in \mathbb{R}^d\) is the resulting dense embedding.

Distance & Similarity Metrics

The semantic relationship between two embedded vectors \(\mathbf{u}, \mathbf{v} \in \mathbb{R}^d\) is quantified using geometric metrics in the embedding space:

• Dot Product: $$\langle \mathbf{u}, \mathbf{v} \rangle = \sum_{k=1}^d u_k v_k$$ Measures unnormalized directional alignment and vector magnitude.
• Cosine Similarity: $$\cos(\theta) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|_2 \|\mathbf{v}\|_2}$$ Measures the cosine of the angle between vectors, normalizing for vector length. Widely preferred in text retrieval because document length variations do not distort thematic alignment.
• Euclidean Distance (\(L_2\)): $$\|\mathbf{u} - \mathbf{v}\|_2 = \sqrt{\sum_{k=1}^d (u_k - v_k)^2}$$ For \(L_2\)-normalized vectors (\(\|\mathbf{u}\|_2 = \|\mathbf{v}\|_2 = 1\)), Euclidean distance relates directly to cosine similarity: \(\|\mathbf{u} - \mathbf{v}\|_2^2 = 2 - 2\cos(\theta)\).

Linear Substructure & Arithmetic

A seminal discovery in distributed representations is that continuous vector spaces encode analogical relationships as consistent directional offsets [3]. Relationships such as gender, grammatical tense, and capital-country associations emerge as stable translation vectors:

This linear property demonstrates that training objectives based on word co-occurrence implicitly perform matrix factorization on point-wise mutual information (PMI) matrices [4].

3. Architectural Evolution

Static Word Embeddings

Early distributed representation frameworks produced static lookup tables where each vocabulary token mapped to a single fixed vector regardless of context:

• Word2Vec (2013): Mikolov et al. introduced Continuous Bag-of-Words (CBOW) to predict target words from context windows, and Continuous Skip-gram to predict surrounding context from target words using negative sampling [3].
• GloVe (2014): Pennington et al. trained vectors directly on global log-bilinear word co-occurrence statistics across whole corpora, combining the advantages of global matrix factorization with local window methods [5].
• FastText (2017): Bojanowski et al. extended embeddings to character \(n\)-grams, allowing models to generate out-of-vocabulary representations for unseen morphological variants [6].

Contextual Sequence Embeddings

Static embeddings fail on polysemy: the token "bank" received the exact same vector in "river bank" and "investment bank." Modern deep architectures produce dynamically contextualized embeddings where each token's vector is a function of the entire sequence:

• ELMo (2018): Bi-directional LSTMs generating layered representations based on sequence context [7].
• Transformer Encoders (BERT, 2019): Bidirectional multi-head self-attention generating deep, dynamically weighted token representations [8].
• Sentence Transformers (SBERT, 2019): Siamese neural networks trained with InfoNCE and multiple-negatives ranking loss to yield semantically meaningful whole-sentence pooling vectors [9].

Positional Encodings

Because the self-attention mechanism is inherently permutation-invariant, positional information must be added directly into token representations [10]:

• Absolute Sinusoidal Embeddings: Fixed frequencies using trigonometric functions across dimensions (original Transformer) [10].
• Rotary Position Embeddings (RoPE): Rotates query and key vectors in complex 2D subspaces based on token index, preserving relative distance decay without parameter inflation [11].

4. Geometric Pathologies: Anisotropy & Degeneration

A persistent challenge in deep language model representation is the anisotropy problem [12]. Rather than utilizing the full capacity of the \(d\)-dimensional hypersphere uniformly, learned token representations tend to collapse into a narrow, cone-shaped subspace. Consequently, arbitrary pairs of unrelated tokens often exhibit high positive cosine similarities (e.g. \(\cos > 0.7\)).

Remediations in modern embedding pipelines include:

• Mean-centering and whitening: Subtracting the empirical mean vector \(\boldsymbol{\mu}\) and transforming vectors by the inverse covariance matrix to restore isotropic distribution.
• Contrastive regularization: Incorporating InfoNCE losses that explicitly push negative pairs apart across the full geometric sphere during fine-tuning.
• Matryoshka Representation Learning (MRL): Structuring embeddings so that truncating a 1536-dimension vector down to 256 or 512 dimensions preserves the highest-variance semantic information with minimal recall loss.

5. Production Retrieval & Vector Indexing

In production applications such as Retrieval-Augmented Generation (RAG) and semantic search, systems query datasets containing millions or billions of embedding vectors. Exact \(k\)-nearest neighbor search via brute-force linear scanning requires \(O(N \cdot d)\) operations per query, which is computationally prohibitive at scale.

Production infrastructures deploy Approximate Nearest Neighbor (ANN) indexing structures:

• Hierarchical Navigable Small World (HNSW): Multi-layer geometric graphs providing logarithmic search complexity \(O(\log N)\) while preserving high recall [13].
• Inverted File with Product Quantization (IVF-PQ): Clusters vector space into Voronoi cells and quantizes sub-vectors into compressed codebooks, reducing RAM requirements by 80% to 95%.