← Reference · Nestor G Pestelos Jr

Reference Document

Deep Neural Networks

A citable reference on Deep Neural Networks (DNNs): multi-layer architecture, non-linear activation functions, backpropagation via the chain rule, and hierarchical representation learning.

1. Definition & Mathematical Formulation

A Deep Neural Network (DNN) is an artificial neural network with multiple hidden layers (\(L \ge 2\)) between its input and output layers. Mathematically, a DNN is a nested composite function that maps an input vector \(x \in \mathbb{R}^{d_{in}}\) to an output vector \(y \in \mathbb{R}^{d_{out}}\):

$$y = f(x; \theta) = f_L\Big(f_{L-1}\big(\dots f_1(x; W_1, b_1)\dots; W_{L-1}, b_{L-1}\big); W_L, b_L\Big)$$

where each layer \(l\) computes an affine linear transformation followed by an element-wise non-linear activation function \(\sigma\):

$$h_l = \sigma(W_l h_{l-1} + b_l)$$

2. The Necessity of Non-Linearity

Without non-linear activation functions, any deep composition of linear layers collapses algebraically into a single matrix multiplication:

$$W_3 (W_2 (W_1 x)) = (W_3 W_2 W_1) x = W_{net} x$$

Non-linear activations (e.g. ReLU \(\max(0, x)\), GELU \(x \Phi(x)\), SwiGLU) allow the network to construct complex, non-linear decision boundaries. Under the Universal Approximation Theorem (Cybenko 1989, Hornik 1991), non-linear feedforward networks can approximate any continuous function on compact subsets of \(\mathbb{R}^n\) to arbitrary precision.

3. Hierarchical Representation Learning

The fundamental advantage of depth over width is automatic feature hierarchy extraction:

• Early Layers (Low-Level): Learn generic primitives (e.g., edge detectors and color blobs in vision; subword token stems and syntactic n-grams in language).
• Intermediate Layers (Mid-Level): Combine primitives into structural motifs (e.g., shapes and textures in vision; part-of-speech dependencies and semantic phrases in language).
• Deep Layers (High-Level): Synthesize abstract conceptual representations (e.g., entire object identities in vision; complex pragmatic intent, facts, and code logic in LLMs).

4. Training, Loss & Backpropagation

Deep networks are trained end-to-end via gradient descent by minimizing an objective empirical loss function \(\mathcal{L}(y_{pred}, y_{true})\) (e.g. Cross-Entropy Loss for language modeling).

• Forward Pass: Propagates input data through all \(L\) layers to compute predictions and loss \(\mathcal{L}\).
• Backward Pass (Backpropagation): Evaluates the exact partial derivatives of the loss with respect to all network weights \(\frac{\partial \mathcal{L}}{\partial W_l}\) via reverse-mode automatic differentiation (multivariable chain rule):

  $$\frac{\partial \mathcal{L}}{\partial W_l} = \frac{\partial \mathcal{L}}{\partial h_l} \cdot \frac{\partial h_l}{\partial z_l} \cdot \frac{\partial z_l}{\partial W_l}$$

• Weight Update: Stochastic gradient optimizers with momentum and decoupled weight decay (e.g. AdamW) update parameters \(\theta \leftarrow \theta - \eta \nabla_{\theta}\mathcal{L}\).

5. Major Architectural Families