← Reference · Nestor G Pestelos Jr · Print this page

Artificial Intelligence

Computer Vision

A citable reference on Computer Vision: image formation geometry, spatial filtering, classical feature descriptors, convolutional and Vision Transformer architectures, object detection, segmentation, and 3D scene representation.

1. Image Formation and Low-Level Processing

Computer Vision is an interdisciplinary field of artificial intelligence and computer science focused on developing algorithmic and computational systems that extract, process, analyze, and understand high-level semantic information from digital images and video sequences.[1, 2]

1.1 Pinhole Camera Geometry and Projection

The geometric mapping from a 3D world coordinate point $X = [X_w, Y_w, Z_w, 1]^T$ in homogeneous coordinates to a 2D image pixel coordinate $x = [u, v, 1]^T$ is modeled by the pinhole camera projection matrix $P \in \mathbb{R}^{3 \times 4}$:[2]

$$x \sim P X = K [R \mid t] X$$

where:

$K \in \mathbb{R}^{3 \times 3}$ is the intrinsic calibration matrix containing focal lengths $(f_x, f_y)$, principal point $(c_x, c_y)$, and skew $s$:
$$K = \begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}$$
$[R \mid t] \in \mathbb{R}^{3 \times 4}$ is the extrinsic matrix representing the 3D rotation $R \in \text{SO}(3)$ and translation vector $t \in \mathbb{R}^3$ transforming world coordinates into the camera reference frame.

1.2 Digital Image Representations

A discrete digital image is represented as a multidimensional array (tensor) $I \in \mathbb{R}^{H \times W \times C}$, where $H$ is height, $W$ is width, and $C$ is channel depth (e.g. $C=1$ for grayscale luminance, $C=3$ for RGB color spaces, or $C > 3$ for multispectral imagery). Continuous intensity signals $f(x, y)$ are digitized via spatial sampling on a pixel grid and radiometric quantization (typically 8-bit integer values in $[0, 255]$ or 32-bit floating-point numbers in $[0, 1]$).

1.3 Linear Spatial Filtering and 2D Convolutions

Linear spatial filtering computes a weighted linear combination of neighboring pixels. For an image $I$ and a discrete kernel $K \in \mathbb{R}^{(2k+1) \times (2k+1)}$, the 2D discrete convolution is:[1]

$$(I * K)(x, y) = \sum_{u=-k}^k \sum_{v=-k}^k I(x - u, y - v) K(u, v)$$

The Gaussian filter kernel smooths noise by isotropic spatial averaging with standard deviation $\sigma$:

$$G_\sigma(x, y) = \frac{1}{2\pi \sigma^2} \exp\left( -\frac{x^2 + y^2}{2\sigma^2} \right)$$

1.4 Edge and Gradient Detection

Image edges correspond to local discontinuities in pixel intensity, computed via the spatial gradient vector $\nabla I = \left[ \frac{\partial I}{\partial x}, \frac{\partial I}{\partial y} \right]^T$.

Sobel Operators: Approximate first horizontal and vertical derivatives using separable $3 \times 3$ difference kernels:
$$K_x = \begin{bmatrix} -1 & 0 & +1 \\ -2 & 0 & +2 \\ -1 & 0 & +1 \end{bmatrix}, \quad K_y = \begin{bmatrix} -1 & -2 & -1 \\ 0 & 0 & 0 \\ +1 & +2 & +1 \end{bmatrix}$$
Canny Edge Detector (Canny, 1986): A multi-stage optimal edge detection algorithm comprising: (1) Gaussian smoothing, (2) gradient magnitude and angle computation, (3) non-maximum suppression (thinning edges to 1-pixel width), and (4) hysteresis thresholding with dual high/low bounds.[3]

2. Classical Feature Extraction and Geometric Vision

2.1 Interest Points and Local Descriptors

Classical computer vision identifies distinctive, repeatable local image regions invariant to geometric scale, rotation, and illumination changes:

Harris Corner Detector (Harris & Stephens, 1988): Analyzes the second-moment structure tensor $M = \sum_{(x,y)} w(x,y) \begin{bmatrix} I_x^2 & I_x I_y \\ I_x I_y & I_y^2 \end{bmatrix}$ and evaluates cornerness score $R = \det(M) - k \, \text{Tr}(M)^2$.[4]
SIFT (Scale-Invariant Feature Transform, Lowe 2004): Detects scale-space extrema across Difference-of-Gaussian (DoG) octaves and constructs a 128-dimensional local gradient orientation histogram descriptor.[5]
HOG (Histogram of Oriented Gradients, Dalal & Triggs 2005): Computes dense grids of normalized local gradient histograms, serving as the foundational descriptor for classical pedestrian detection.[6]

2.2 Epipolar Geometry and Multi-View Stereo

Given two calibrated views of a 3D scene, matching pixel coordinates $x$ and $x'$ satisfy the epipolar constraint defined by the Fundamental Matrix $F \in \mathbb{R}^{3 \times 3}$:[2]

$$x'^T F x = 0, \quad \text{where } F = K'^{-T} [t]_\times R \, K^{-1}$$

In normalized camera coordinates, the Essential Matrix $E = [t]_\times R$ satisfies $x_{\text{norm}}'^T E x_{\text{norm}} = 0$, enabling 3D point triangulation, Structure from Motion (SfM), and visual Simultaneous Localization and Mapping (SLAM).

3. Deep Learning Architectures for Vision

3.1 Convolutional Neural Networks (CNNs)

Convolutional Neural Networks exploit two structural inductive biases inherent to natural images: local receptive fields and translation equivariance ($f(T_g(x)) = T_g(f(x))$).[7]

A convolutional layer applies learnable 3D filter banks $W \in \mathbb{R}^{C_{\text{out}} \times C_{\text{in}} \times k_h \times k_w}$ followed by non-linear activations (e.g. ReLU $\max(0, z)$) and spatial downsampling pooling operations (Max Pooling, Average Pooling, or strided convolutions). AlexNet (Krizhevsky et al., 2012) demonstrated the breakthrough capability of deep GPU-trained CNNs on the ImageNet benchmark.[8]

3.2 Residual Networks (ResNet)

As networks grow deeper, gradient vanishing and degradation hinder optimization. ResNet (He et al., 2016) introduces identity skip connections that reformulate layer blocks to learn residual mappings $\mathcal{F}(x)$:[9]

$$y = \mathcal{F}(x, \{W_i\}) + x$$

Residual connections ensure unobstructed gradient backpropagation directly through the identity pathway, enabling stable training of architectures spanning hundreds of layers.

3.3 Vision Transformers (ViT)

Vision Transformers (Dosovitskiy et al., 2020) eliminate convolutional inductive biases in favor of standard Transformer encoders.[10] An image $I \in \mathbb{R}^{H \times W \times C}$ is divided into $N = \frac{HW}{P^2}$ non-overlapping patches of size $P \times P$. Each patch is flattened and linearly projected into a $D$-dimensional embedding:

$$z_0 = \left[ x_{\text{class}}; \, x_p^1 E; \, x_p^2 E; \dots; \, x_p^N E \right] + E_{\text{pos}}, \quad E \in \mathbb{R}^{(P^2 C) \times D}, \quad E_{\text{pos}} \in \mathbb{R}^{(N+1) \times D}$$

Global multi-head self-attention computes pairwise patch interactions across the entire image from the first layer, achieving superior scaling efficiency when pre-trained on large-scale datasets (e.g. JFT-300M, LAION-5B).

4. Core Visual Tasks and Objective Formulations

4.1 Image Classification

Image classification maps an image $I$ to a categorical distribution over $C$ classes. Models are trained by minimizing the multiclass cross-entropy loss over one-hot target vectors $y$:

$$\mathcal{L}_{\text{CE}} = -\sum_{c=1}^C y_c \log \hat{p}_c, \quad \text{where } \hat{p}_c = \frac{\exp(z_c)}{\sum_{j=1}^C \exp(z_j)}$$

4.2 Object Detection (YOLO, Faster R-CNN, DETR)

Object detection predicts bounding box coordinates $b = [x, y, w, h]$ and associated class labels for all object instances in an image.

Paradigm	Representative Architectures	Mechanism & Loss Formulation
Two-Stage Detectors	Faster R-CNN, Cascade R-CNN	Stage 1 generates candidate proposals via Region Proposal Networks (RPN); Stage 2 performs RoIAlign pooling, box refinement, and classification.
One-Stage Detectors	YOLO (v1–v11), SSD, RetinaNet	Direct dense regression of bounding boxes and class probabilities across feature pyramid grids using Focal Loss: $\text{FL}(p_t) = -\alpha_t (1 - p_t)^\gamma \log(p_t)$.[11, 12]
Transformer Detectors	DETR, Deformable DETR	End-to-end set prediction using learned object queries and bipartite matching loss solved via the Hungarian algorithm.[13]

4.3 Semantic and Instance Segmentation

Semantic Segmentation: Assigns a categorical class label to every pixel without distinguishing separate object instances. Architectures utilize encoder-decoder pathways with skip connections (e.g. U-Net, DeepLabv3+ with atrous spatial pyramid pooling).[14]
Instance Segmentation: Detects and delineates a distinct pixel mask for each individual object instance (e.g. Mask R-CNN adding a binary mask prediction branch to Faster R-CNN).[15]
Foundation Segmentation: Promptable zero-shot segmentation across arbitrary images (e.g. Segment Anything Model / SAM).[16]

4.4 3D Vision, NeRFs, and Gaussian Splatting

Modern computer vision reconstructs photorealistic, view-dependent 3D scenes from multi-view 2D image collections:

Neural Radiance Fields (NeRF, Mildenhall et al. 2020): Represents continuous 3D scenes as an implicit neural network function $F_\Theta(x, y, z, \theta, \phi) \to (\sigma, c)$ mapping 3D coordinates and viewing directions to volume density $\sigma$ and RGB color $c$, rendered via differentiable numerical ray marching.[17]
3D Gaussian Splatting (Kerbl et al. 2023): Represents scenes as collections of 3D anisotropic Gaussians parameterized by center $\mu$, covariance $\Sigma = R S S^T R^T$, opacity $\alpha$, and spherical harmonic color coefficients, rendered via fast tile-based rasterization at real-time frame rates.[18]

5. Evaluation Metrics in Computer Vision

Task	Standard Metrics	Mathematical Definition / Interpretation
Classification	Top-1 / Top-5 Accuracy	Fraction of test samples where ground-truth label matches top-1 or is within top-5 predicted logits.
Detection	Mean Average Precision (mAP)	Area under the Precision-Recall curve averaged across classes at specified Intersection over Union (IoU) thresholds (e.g. $\text{mAP}_{50}$, $\text{mAP}_{50:95}$): $$\text{IoU} = \frac{\|\text{Box}_A \cap \text{Box}_B\|}{\|\text{Box}_A \cup \text{Box}_B\|}$$
Segmentation	Mean IoU (mIoU), Dice Score	$\text{mIoU} = \frac{1}{C} \sum_{c=1}^C \frac{\|P_c \cap G_c\|}{\|P_c \cup G_c\|}$, quantifying pixel overlap between prediction $P_c$ and ground truth $G_c$.
3D Synthesis	PSNR, SSIM, LPIPS	Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS).

6. References

^ Richard Szeliski, Computer Vision: Algorithms and Applications, 2nd ed. (Springer, 2022). DOI: 10.1007/978-3-030-34372-9.
^ Richard Hartley and Andrew Zisserman, Multiple View Geometry in Computer Vision, 2nd ed. (Cambridge University Press, 2004).
^ John Canny, "A Computational Approach to Edge Detection," IEEE Transactions on Pattern Analysis and Machine Intelligence 8(6), 679–698 (1986). DOI: 10.1109/TPAMI.1986.4767851.
^ Chris Harris and Mike Stephens, "A Combined Corner and Edge Detector," Procedings of the Alvey Vision Conference, 147–151 (1988). DOI: 10.5244/C.2.23.
^ David G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," International Journal of Computer Vision 60(2), 91–110 (2004). DOI: 10.1023/B:VISI.0000029664.99615.94.
^ Navneet Dalal and Bill Triggs, "Histograms of oriented gradients for human detection," IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) 1, 886–893 (2005). DOI: 10.1109/CVPR.2005.177.
^ Yann LeCun, Bernhard Boser, John S. Denker, et al., "Backpropagation Applied to Handwritten Zip Code Recognition," Neural Computation 1(4), 541–551 (1989). DOI: 10.1162/neco.1989.1.4.541.
^ Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," Advances in Neural Information Processing Systems 25 (NeurIPS 2012). URL: NeurIPS Proceedings.
^ Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, "Deep Residual Learning for Image Recognition," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778 (2016). DOI: 10.1109/CVPR.2016.90.
^ Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale," International Conference on Learning Representations (ICLR 2021), arXiv:2010.11929 (2020). DOI: 10.48550/arXiv.2010.11929.
^ Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779–788 (2016). DOI: 10.1109/CVPR.2016.91.
^ Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, "Focal Loss for Dense Object Detection," IEEE International Conference on Computer Vision (ICCV), 2980–2988 (2017). DOI: 10.1109/ICCV.2017.324.
^ Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko, "End-to-End Object Detection with Transformers," European Conference on Computer Vision (ECCV), 213–229 (2020). DOI: 10.1007/978-3-030-58452-8_13.
^ Olaf Ronneberger, Philipp Fischer, and Thomas Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation," Medical Image Computing and Computer-Assisted Intervention (MICCAI), 234–241 (2015). DOI: 10.1007/978-3-319-24574-4_28.
^ Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, "Mask R-CNN," IEEE International Conference on Computer Vision (ICCV), 2961–2969 (2017). DOI: 10.1109/ICCV.2017.322.
^ Alexander Kirillov, Eric Mintun, Nikhila Ravi, et al., "Segment Anything," IEEE/CVF International Conference on Computer Vision (ICCV), 4015–4026 (2023). DOI: 10.1109/ICCV51070.2023.00371.
^ Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis," European Conference on Computer Vision (ECCV), 405–421 (2020). DOI: 10.1007/978-3-030-58452-8_24.
^ Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis, "3D Gaussian Splatting for Real-Time Radiance Field Rendering," ACM Transactions on Graphics 42(4), 139:1–139:14 (2023). DOI: 10.1145/3592433.