Tensor (Mathematics and Computing)

A tensor is an algebraic object that maps geometric and physical vectors to scalars in a multilinear fashion, or, in computational systems, a multidimensional array of numerical values indexed across an arbitrary number of discrete dimensions.[1, 2] While differential geometry and theoretical physics treat tensors as multilinear maps governed by strict coordinate-transformation laws, modern computing platforms like PyTorch, TensorFlow, and dedicated accelerator silicon treat tensors as contiguous or strided memory buffers equipped with dimension metadata. Tensors serve as the fundamental data substrate for neural network representations, deep learning parameter storage, and distributed accelerator workloads.

1. First Principles: Multilinear Algebra and Tensor Products

Modern tensor theory rests on the foundations of multilinear algebra. Rather than beginning with coordinate tables or index grids, modern mathematics defines a tensor by its coordinate-free algebraic behavior when acting on vector spaces and their linear duals.[1]

1.1 Dual Spaces and Multilinear Maps

Let \(V\) be a finite-dimensional vector space over a field \(\mathbb{F}\) (typically \(\mathbb{R}\) or \(\mathbb{C}\)). The dual space \(V^*\) consists of all linear functionals \(f: V \to \mathbb{F}\). Given vector spaces \(V_1, V_2, \dots, V_k\), a function

is multilinear if it is linear in each argument independently when holding all other arguments fixed. That is, for any index \(i \in \{1, \dots, k\}\), scalars \(\alpha, \beta \in \mathbb{F}\), and vectors \(u, v \in V_i\):

A tensor of type \((r, s)\) on \(V\) is formally defined as a multilinear map that takes \(r\) dual vectors and \(s\) vectors as inputs to produce a scalar:

The integer \(r\) denotes the contravariant order, \(s\) denotes the covariant order, and the sum \(r + s\) defines the total rank (or valence) of the tensor.[2]

1.2 The Universal Property of the Tensor Product

Multilinear maps with fixed domain spaces and codomain form a vector space under pointwise addition and scalar multiplication. A bilinear map is generally not jointly linear on the Cartesian product \(V \times W\). The tensor product \(V \otimes W\) lets each such map factor through a unique linear map.[9]

Formally, the tensor product of vector spaces \(V\) and \(W\) is a vector space \(V \otimes W\) accompanied by a bilinear map \(\tau: V \times W \to V \otimes W\), denoted \(\tau(v, w) = v \otimes w\), satisfying the universal property: for every vector space \(U\) and every bilinear map \(B: V \times W \to U\), there exists a unique linear map \(\tilde{B}: V \otimes W \to U\) such that \(B = \tilde{B} \circ \tau\).

If \(\{e_i\}\) forms a basis for \(V\) of dimension \(m\), and \(\{f_j\}\) forms a basis for \(W\) of dimension \(n\), then the set of simple tensors \(\{e_i \otimes f_j\}\) forms a basis for \(V \otimes W\) of dimension \(m \cdot n\). Any arbitrary tensor \(T \in V \otimes W\) decomposes into a linear combination of these basis elements:

1.3 Tensor Type, Covariance, and Contravariance

Tensors are classified according to how their arguments relate to vectors and linear functionals:

• Type (0, 0): Scalars in the base field \(\mathbb{F}\). Invariant under coordinate system transformations.
• Type (1, 0): Contravariant vectors (\(v \in V\)). Linear maps on the dual space \(V^* \to \mathbb{F}\).
• Type (0, 1): Covariant vectors or covectors (\(\omega \in V^*\)). Linear functionals on \(V \to \mathbb{F}\).
• Type (1, 1): Linear operators \(A: V \to V\), isomorphic to \(V \otimes V^*\). Matrices mapping vectors to vectors.
• Type (0, 2): Bilinear forms on \(V \times V \to \mathbb{F}\), such as the metric tensor \(g_{\mu\nu}\) in differential geometry and general relativity.
• Type (1, 3): The Riemann curvature tensor \(R^\rho_{\ \sigma\mu\nu}\), measuring the non-commutativity of covariant derivatives along smooth manifolds.

2. Coordinate Systems and Transformation Laws

When coordinates are introduced on an underlying manifold or vector space, the components of a tensor transform in a specific manner under a smooth change of coordinates.[2]

2.1 Change of Basis and Index Notation

Consider a coordinate change from \(\{x^\mu\}\) to \(\{\tilde{x}^\alpha\}\). The Jacobian matrix of partial derivatives governs the local transformation:

A tensor of type \((r, s)\) with components \(T^{\mu_1 \dots \mu_r}_{\ \nu_1 \dots \nu_s}\) transforms to the new coordinate basis according to the general multilinear transformation law:

Contravariant indices (superscripts) transform via the forward Jacobian matrix \(\frac{\partial \tilde{x}^\alpha}{\partial x^\mu}\), opposing the transformation of basis vectors. Covariant indices (subscripts) transform via the inverse Jacobian matrix \(\frac{\partial x^\nu}{\partial \tilde{x}^\beta}\), in the same manner as basis vectors.

2.2 Einstein Summation Convention

Introduced by Albert Einstein in 1916, index summation notation omits explicit summation signs when an index appears once as an upper index and once as a lower index within a single term:

Indices that appear matched are dummy indices, representing summation. Unmatched indices are free indices, indicating the tensorial rank and dimension of the resulting expression. Contracting a contravariant index with a covariant index reduces the total tensor order by 2 (reducing an \((r, s)\) tensor to an \((r-1, s-1)\) tensor).

2.3 Geometric Invariance

While numerical component representations change based on the choice of basis, the underlying tensor object remains coordinate-independent. In differential geometry, tensor fields assign a multilinear map to each tangent space \(T_p M\) and cotangent space \(T_p^* M\) of a smooth manifold \(M\). If all components of a tensor vanish in one coordinate frame, the transformation laws dictate that they vanish in every coordinate frame, making tensor equations the natural language for physical laws (general covariance).[2]

3. Computational Tensors and Multidimensional Arrays

In computer science, software engineering, and machine learning, the word tensor carries a pragmatic, operational definition distinct from its mathematical namesake.[3, 4]

3.1 The Terminology Gap: Geometric Tensor vs. Data Array

A computational tensor is an \(N\)-dimensional numerical array characterized by:

• Data Type (dtype): The element primitive (e.g., FP32, BF16, FP8, INT8).
• Shape (dimensions): A tuple of non-negative integers \((d_0, d_1, \dots, d_{N-1})\) indicating axis lengths.
• Order / Rank: The number of axes \(N\) (e.g., scalar = 0D, vector = 1D, matrix = 2D, volume/batch = 3D+).

In pure mathematics, an arbitrary 3D array of numbers is not necessarily a tensor unless it satisfies coordinate transformation laws under a change of basis. In computing frameworks, any multidimensional table of numerical values is termed a tensor regardless of variance or coordinate behavior.

3.2 Strides, Offsets, and Contiguous Memory Layouts

Let \((s_0, s_1, \dots, s_{N-1})\) be a stride tuple measured in elements, and let itemsize be the number of bytes per element. With zero-based indices and byte_offset measured from the storage start to the first logical element, the byte offset of an element is:

For a C-contiguous int32 array shaped (2, 3, 4), the element strides are (12, 4, 1). Index (1, 1, 1) lies 4 × (12 + 4 + 1) = 68 bytes from its first element. NumPy reports strides in bytes: (48, 16, 4) for this array.[10]

The two primary ordering conventions for dense arrays, using element strides, are:

• Row-Major (C-contiguous): The last dimension varies fastest in memory. For a shape \((d_0, d_1, d_2)\), the strides are \(s_2 = 1\), \(s_1 = d_2\), and \(s_0 = d_1 \cdot d_2\). This is the default in PyTorch, NumPy, and C.
• Column-Major (Fortran-contiguous): The first dimension varies fastest in memory. Strides are \(s_0 = 1\), \(s_1 = d_0\), and \(s_2 = d_0 \cdot d_1\). Used in MATLAB, Fortran, and standard BLAS kernels.

3.3 Zero-Copy Views vs. Memory Allocations

Modern machine learning frameworks decouple logical shape metadata from underlying memory storage:[4]

• Storage Buffer: A flat, 1D array of raw allocated bytes on CPU RAM, GPU High-Bandwidth Memory (HBM), or TPU SRAM.
• Tensor View: Metadata containing a pointer to the storage, an offset, a shape tuple, and a stride tuple.

Operations such as slicing, transposing (permuting dimensions), expanding (broadcasting), and squeezing manipulate only shape and stride tuples. These operations execute in \(O(1)\) constant time without copying underlying bytes. A tensor becomes non-contiguous when its strides deviate from standard row-major progression (for instance, after a transpose). Calling operations that expect contiguous bytes requires an explicit copy step (such as PyTorch's .contiguous()) to reorder memory sequentially.

4. Tensor Operations and Decompositions

High-order data collections quickly become intractable due to exponential growth in parameters. Tensor decompositions generalize matrix factorizations (such as SVD and Eigendecomposition) to higher dimensions.[5]

4.1 Tensor Contractions and Outer Products

Fundamental operations define transformations on multidimensional arrays:

• Outer Product (\(\otimes\)): Given \(\mathcal{A} \in \mathbb{R}^{I_1 \times \dots \times I_P}\) and \(\mathcal{B} \in \mathbb{R}^{J_1 \times \dots \times J_Q}\), their outer product produces a tensor of rank \(P+Q\) with entries \((\mathcal{A} \otimes \mathcal{B})_{i_1 \dots i_P j_1 \dots j_Q} = \mathcal{A}_{i_1 \dots i_P} \mathcal{B}_{j_1 \dots j_Q}\).
• Tensor Contraction: Summing over matching indices across dimensions. General Matrix Multiply (GEMM) is a single contraction between two 2D tensors over an inner dimension. Multi-head self-attention computes batched contractions across sequence, head, and hidden dimensions.

4.2 Canonical Polyadic (CP) Decomposition

The Canonical Polyadic (CP) decomposition (also known as CANDECOMP/PARAFAC) factorizes a tensor \(\mathcal{X} \in \mathbb{R}^{I \times J \times K}\) into a sum of component rank-1 tensors:[5]

where \(a_r \in \mathbb{R}^I\), \(b_r \in \mathbb{R}^J\), \(c_r \in \mathbb{R}^K\), and \(R\) is the chosen number of components in the approximation. The CP rank of \(\mathcal{X}\) is the minimum number of rank-1 terms needed for exact equality. Rank-1 means an outer product of vectors. CP rank differs from the number of axes, called order or rank in Section 3.1. Unlike matrix rank, determining the exact CP rank of a general 3D tensor is an NP-hard problem, and the best rank-\(R\) approximation may not always exist due to non-closed solution spaces.

4.3 Tucker Decomposition and Tensor Trains

Tucker Decomposition: A higher-order form of Principal Component Analysis (PCA). It decomposes a tensor into a small dense core tensor \(\mathcal{G} \in \mathbb{R}^{P \times Q \times R}\) multiplied by orthogonal factor matrices along each mode:

Tensor Train (TT) Decomposition: Mitigates the curse of dimensionality for very high orders by factorizing an \(N\)-way tensor into a linear chain of interconnected 3D core tensors. TT decomposition compresses parameter counts from exponential \(O(d^N)\) to polynomial \(O(N \cdot d \cdot r^2)\), enabling simulation of quantum many-body states and low-rank compression of neural network weights.[6]

5. Tensors in Deep Learning Systems and Hardware

In modern artificial intelligence, neural networks are constructed as computational graphs where nodes represent mathematical transformations and edges represent flowing data tensors.[3]

5.1 Automatic Differentiation and Execution Graphs

Deep learning frameworks compute gradients across thousands of tensor variables using reverse-mode automatic differentiation (backpropagation). During the forward pass, frameworks record operations in a Directed Acyclic Graph (DAG). Each tensor node tracks:

• The underlying forward data buffer.
• The backward gradient tensor of identical shape (\(\nabla_W \mathcal{L}\)).
• A reference to the generating function (e.g., AddBackward, MmBackward).

During the backward pass, vector-Jacobian products (VJPs) traverse the graph in reverse topological order, accumulating partial derivatives directly into parameter gradient buffers.

5.2 Systolic Arrays and Tensor Cores

General-purpose CPUs execute sequential scalar or small vector SIMD instructions. Because deep learning workloads are dominated by large matrix multiplications and convolutions, modern AI accelerators incorporate specialized 2D arithmetic matrix hardware:[7]

• Systolic Arrays (Google TPU): A 2D grid of Arithmetic Logic Units (ALUs). Data streams continuously across rows and columns without returning intermediate results to register files, computing matrix multiplications with high energy efficiency.
• Tensor Cores (NVIDIA Volta, Hopper, Blackwell): Specialized hardware execution units that perform a fused multiply-add on small matrix tiles (e.g., \(4 \times 4\) or \(16 \times 16\)) in a single clock cycle, operating on low-precision formats (FP16, BF16, FP8, FP4) and accumulating in FP32.

5.3 Tensor Parallelism and Model Sharding

When large language model parameter tensors exceed the memory capacity of a single GPU (e.g., 70B+ parameters), distributed frameworks split individual weight tensors across multiple accelerators using Tensor Parallelism (Megatron-LM):[8]

• Column Parallel Linear Layers: The weight matrix \(W\) is partitioned along columns: \(W = [W_1 \mid W_2]\). Accelerators compute local projections \(Y_i = X W_i\) in parallel.
• Row Parallel Linear Layers: The weight matrix \(W\) is partitioned along rows. Each GPU multiplies its partitioned activation slice by the row slice and performs an All-Reduce (Sum) communication collective across NVLink interconnects to produce the final output tensor.