← Reference · Nestor G Pestelos Jr · Print this page

Machine Learning · Computer Science

Neural Networks

Reference entry · last updated August 29, 2026

Neural networks (also termed artificial neural networks or ANNs) are computational learning models composed of interconnected processing units called artificial neurons.[1] Inspired by biological neural networks, ANNs map complex, non-linear relationships between inputs and outputs by adjusting connection weights through gradient optimization on training data.[2]

Historical foundations

The mathematical study of artificial neural networks developed across four distinct phases:

Threshold Logic (1943): Warren McCulloch and Walter Pitts formulated the first mathematical model of a biological neuron as a binary threshold device computing logical propositions (AND, OR, NOT).[1]
The Perceptron (1958): Frank Rosenblatt introduced the single-layer perceptron, adding real-valued learnable synaptic weights updated iteratively on classification errors.[3]
Linear Separability Bounds (1969): Marvin Minsky and Seymour Papert demonstrated that single-layer perceptrons cannot solve linearly non-separable functions such as the exclusive-OR (XOR) logic gate.[4]
Multilayer Backpropagation (1986): David Rumelhart, Geoffrey Hinton, and Ronald Williams popularized the generalized delta rule (backpropagation), enabling efficient end-to-end gradient descent training through hidden layers with non-linear activations.[5]

Artificial neuron mathematical model

An artificial neuron receives an input vector \(\mathbf{x} = [x_1, x_2, \dots, x_d]^T \in \mathbb{R}^d\), computes a weighted linear combination offset by a scalar bias \(b \in \mathbb{R}\), and applies an activation function \(\sigma\):[2]

The pre-activation net input \(z\) is defined as:

\[z = \mathbf{w}^T \mathbf{x} + b = \sum_{i=1}^{d} w_i x_i + b\]

The post-activation output scalar \(a\) is:

\[a = \sigma(z) = \sigma\left(\mathbf{w}^T \mathbf{x} + b\right)\]

The weight vector \(\mathbf{w}\) determines the orientation and slope of the decision boundary, while the bias \(b\) translates the boundary away from the coordinate origin.

Activation functions

Without non-linear activation functions, a neural network with arbitrary depth collapses into a single linear transformation \(\mathbf{y} = \mathbf{W}_{\text{total}} \mathbf{x} + \mathbf{b}_{\text{total}}\). Non-linear activations introduce the curvature required to separate complex data manifolds.[2]

Activation Function	Mathematical Formulation	Output Range	Key Characteristic
Sigmoid (Logistic)	\(\sigma(z) = \frac{1}{1 + e^{-z}}\)	\((0, 1)\)	Interpretable probability; saturates gradients for \(\|z\| \gg 0\).
Hyperbolic Tangent (\(\tanh\))	\(\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}\)	\((-1, 1)\)	Zero-centered output; reduces directional gradient bias.
ReLU (Rectified Linear Unit)	\(f(z) = \max(0, z)\)	\([0, \infty)\)	Constant unit derivative for \(z > 0\); eliminates vanishing gradients.
Leaky ReLU	\(f(z) = \max(\alpha z, z), \; \alpha \approx 0.01\)	\((-\infty, \infty)\)	Prevents dead neurons by sustaining non-zero gradient for \(z < 0\).
Softmax	\(\text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}}\)	\((0, 1), \; \sum a_i = 1\)	Normalizes a logits vector into a valid probability distribution.

Multilayer architectures

A Multilayer Perceptron (MLP) structures neurons into sequential layers: an input layer, one or more intermediate hidden layers, and an output layer.[2]

For layer \(l \in \{1, 2, \dots, L\}\) with weight matrix \(\mathbf{W}^{[l]} \in \mathbb{R}^{n_l \times n_{l-1}}\) and bias vector \(\mathbf{b}^{[l]} \in \mathbb{R}^{n_l}\), the vectorized forward pass operates as:

\[\mathbf{z}^{[l]} = \mathbf{W}^{[l]} \mathbf{a}^{[l-1]} + \mathbf{b}^{[l]}\] \[\mathbf{a}^{[l]} = \sigma^{[l]}\left(\mathbf{z}^{[l]}\right)\]

where \(\mathbf{a}^{[0]} = \mathbf{x}\) is the input vector and \(\hat{\mathbf{y}} = \mathbf{a}^{[L]}\) is the network prediction.

Learning and backpropagation

Training a neural network consists of minimizing a scalar loss function \(\mathcal{L}(\hat{\mathbf{y}}, \mathbf{y})\) over the training dataset through iterative gradient descent updates.[5]

Canonical loss functions

Mean Squared Error (Regression): \[\mathcal{L}_{\text{MSE}} = \frac{1}{2n} \sum_{i=1}^n \left\| \hat{\mathbf{y}}_i - \mathbf{y}_i \right\|^2\]
Categorical Cross-Entropy (Multi-class Classification): \[\mathcal{L}_{\text{CE}} = -\sum_{k=1}^K y_k \log \hat{y}_k\]

The chain rule and error backpropagation

Backpropagation computes exact partial derivatives of \(\mathcal{L}\) with respect to all internal parameters by applying the calculus chain rule in reverse topological order from output layer \(L\) back to input layer \(1\):[5]

For the output layer \(L\), the error vector \(\boldsymbol{\delta}^{[L]}\) is:

\[\boldsymbol{\delta}^{[L]} = \frac{\partial \mathcal{L}}{\partial \mathbf{z}^{[L]}} = \nabla_{\mathbf{a}^{[L]}} \mathcal{L} \odot {\sigma^{[L]}}'\left(\mathbf{z}^{[L]}\right)\]

For each preceding hidden layer \(l = L-1, L-2, \dots, 1\), error propagates backward:

\[\boldsymbol{\delta}^{[l]} = \left( (\mathbf{W}^{[l+1]})^T \boldsymbol{\delta}^{[l+1]} \right) \odot {\sigma^{[l]}}'\left(\mathbf{z}^{[l]}\right)\]

The parameter gradients are then computed via outer products:

\[\frac{\partial \mathcal{L}}{\partial \mathbf{W}^{[l]}} = \boldsymbol{\delta}^{[l]} (\mathbf{a}^{[l-1]})^T, \quad \frac{\partial \mathcal{L}}{\partial \mathbf{b}^{[l]}} = \boldsymbol{\delta}^{[l]}\]

Parameters update according to learning rate \(\eta > 0\):

\[\mathbf{W}^{[l]} \leftarrow \mathbf{W}^{[l]} - \eta \frac{\partial \mathcal{L}}{\partial \mathbf{W}^{[l]}}, \quad \mathbf{b}^{[l]} \leftarrow \mathbf{b}^{[l]} - \eta \frac{\partial \mathcal{L}}{\partial \mathbf{b}^{[l]}}\]

Universal Approximation Theorem

The Universal Approximation Theorem, proven by George Cybenko (1989) for sigmoidal activations and generalized by Kurt Hornik (1991) for arbitrary non-constant bounded continuous activations, establishes the theoretical representational capacity of neural networks:[6, 7]

A standard feedforward neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function \(f: \mathbb{R}^d \to \mathbb{R}^m\) on compact subsets of \(\mathbb{R}^d\) to arbitrary precision \(\epsilon > 0\), given appropriate parameters.

While the theorem proves the existence of a single-layer network capable of representation, it does not guarantee that gradient descent can efficiently learn the weights, nor does it bound the required number of hidden neurons, which can scale exponentially with input dimensionality.

Architectural taxonomy

Specialized neural network architectures incorporate inductive biases tailored to distinct data geometries:[8]

Feedforward Neural Networks (FNN / MLP): Fully connected acyclic graphs used primarily for tabular data, regression, and discrete classification.
Convolutional Neural Networks (CNN): Exploit translation invariance and spatial locality through discrete convolutions, weight sharing, and pooling layers (e.g. LeNet, ResNet).
Recurrent Neural Networks (RNN, LSTM, GRU): Maintain an internal recurrent hidden state \(h_t = f(h_{t-1}, x_t)\) to process sequential time-series and token streams.
Transformer Architectures: Replace recurrence entirely with multi-head self-attention mechanisms, enabling parallel computation across long-range token contexts.[9]
Autoencoders: Bottleneck architectures that learn compressed latent representations by reconstructing their own input signals under dimensionality or sparsity constraints.

Generalization and regularization

Overparameterized neural networks possess sufficient capacity to memorize training labels. Preventing overfitting requires empirical regularization techniques:[2]

Weight Decay (\(L_2\) Regularization): Penalizes large parameter norms by adding \(\frac{\lambda}{2} \|\mathbf{W}\|_2^2\) to the loss objective, driving weights toward smaller magnitudes.
Dropout: Randomly deactivates a fraction \(p\) of neuron activations during each forward training step, preventing co-adaptation of features.[10]
Normalization Layers: Batch Normalization and Layer Normalization stabilize internal covariate shifts and smooth optimization loss landscapes.
Early Stopping: Halts gradient updates once loss on a held-out validation dataset begins to diverge from training loss.

References

^ Warren S. McCulloch & Walter Pitts, "A Logical Calculus of the Ideas Immanent in Nervous Activity," Bulletin of Mathematical Biophysics 5(4), 115–133 (1943). DOI: 10.1007/BF02478259.
^ Ian Goodfellow, Yoshua Bengio, & Aaron Courville, Deep Learning (MIT Press, 2016). Free online: deeplearningbook.org.
^ Frank Rosenblatt, "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain," Psychological Review 65(6), 386–408 (1958). DOI: 10.1037/h0042519.
^ Marvin Minsky & Seymour Papert, Perceptrons: An Introduction to Computational Geometry (MIT Press, 1969).
^ David E. Rumelhart, Geoffrey E. Hinton, & Ronald J. Williams, "Learning Representations by Back-Propagating Errors," Nature 323(6088), 533–536 (1986). DOI: 10.1038/323533a0.
^ George Cybenko, "Approximation by Superpositions of a Sigmoidal Function," Mathematics of Control, Signals and Systems 2(4), 303–314 (1989). DOI: 10.1007/BF02551274.
^ Kurt Hornik, "Approximation Capabilities of Multilayer Feedforward Networks," Neural Networks 4(2), 251–257 (1991). DOI: 10.1016/0893-6080(91)90009-T.
^ Yann LeCun, Yoshua Bengio, & Geoffrey Hinton, "Deep Learning," Nature 521(7553), 436–444 (2015). DOI: 10.1038/nature14539.
^ Ashish Vaswani, Noam Shazeer, Niki Parmar, et al., "Attention Is All You Need," in Advances in Neural Information Processing Systems 30 (NeurIPS 2017). arXiv: 1706.03762.
^ Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, & Ruslan Salakhutdinov, "Dropout: A Simple Way to Prevent Neural Networks from Overfitting," Journal of Machine Learning Research 15(56), 1929–1958 (2014).