Nestor G Pestelos Jr · Reference · Print this page

Pre-Training

Published September 3, 2026 · Deep Learning & Language Modeling

Pre-training is the initial self-supervised phase of machine learning in which a neural network optimizes billions of parameters over massive, uncurated or weakly curated datasets. By predicting masked or subsequent tokens across trillions of text fragments, the model encodes generalized linguistic structure, factual associations, and latent reasoning abstractions prior to task-specific adaptation or alignment.

1. Overview & The Self-Supervised Paradigm

Before modern pre-training, natural language processing relied on supervised learning, where human annotators manually labeled training examples for isolated tasks such as sentiment classification, named entity recognition, or syntactic parsing [1]. This paradigm suffered from severe data bottlenecks: human labeling does not scale to billions of examples, and models trained on narrow tasks fail to generalize beyond their training distribution.

Pre-training replaced supervised human labeling with self-supervision [2]. The supervisory signal is generated directly from the natural structure of the text itself by predicting missing words or calculating the likelihood of upcoming tokens. In this stage, the base model acquires broad world knowledge, statistical syntax, and internal semantic abstractions, acting as a foundational base for fine-tuning.

2. Mathematical Objectives & Architectures

Causal Language Modeling (CLM)

Autoregressive generative models (such as GPT-4, Llama, and Mistral) utilize Causal Language Modeling. Given an arbitrary token sequence \(\mathbf{x} = (x_1, x_2, \dots, x_T)\), the network is trained using maximum likelihood estimation to minimize cross-entropy loss over next-token predictions:

where \(P(x_t \mid x_{

Masked Language Modeling (MLM)

Bidirectional encoder models (such as BERT and RoBERTa) optimize a denoising autoencoder objective [3]. A stochastic subset of tokens \(M \subset \{1, \dots, T\}\) (typically 15%) is corrupted: 80% are replaced with a special [MASK] token, 10% are replaced with an arbitrary random token, and 10% remain unchanged:

Because self-attention is unmasked and bidirectional, MLM hidden states condition on both left and right context simultaneously, making MLM effective for feature extraction, semantic search embeddings, and classification, but inefficient for open-ended text generation.

3. Compute Scaling Laws & Chinchilla Frontier

Pre-training performance follows empirical power laws relating parameter count \(N\), training dataset size in tokens \(D\), and total floating-point operations \(C\) [4]. Total training compute is approximated by:

In 2020, Kaplan et al. at OpenAI posited that model size \(N\) should scale substantially faster than dataset token volume \(D\) as compute increases [4]. In 2022, Hoffmann et al. at DeepMind demonstrated that Kaplan et al. evaluated sub-optimally scheduled learning rates, resulting in severely undertrained models. Fitting the empirical parametric loss surface:

DeepMind established the Chinchilla scaling law: for compute-optimal training, model parameter count \(N\) and training token volume \(D\) must scale in equal proportion (\(\alpha \approx \beta \approx 0.5\)):

Under Chinchilla optimality, a 70-billion-parameter model requires approximately 1.4 trillion tokens. Subsequent open-weights models (such as Llama 3) deliberately overtrain past the Chinchilla compute frontier (e.g. 15 trillion tokens for an 8B model) to minimize downstream inference latency and memory footprints at serving time.

4. Data Curation & Filtering Pipelines

Modern pre-training datasets span 5 to 20 trillion tokens. Raw web crawls (such as Common Crawl) cannot be fed directly to training clusters without extensive multi-stage filtering [5]:

• Text Extraction & Normalization: Removing HTML boilerplate, normalizing Unicode characters, and stripping machine-generated SEO spam.
• Quality Filtering: Classifier-based filtering (such as FastText or small Transformers trained on curated corpora like Wikipedia, textbooks, and peer-reviewed journals) and heuristic filters (word count distributions, symbol-to-word ratios, repetition checks).
• Fuzzy Deduplication: MinHash and Locality-Sensitive Hashing (LSH) algorithms detect and purge near-duplicate documents. Deduplication prevents verbatim memorization and accelerates convergence.
• Privacy & Safety: Automated regex and entity-detection filters mask personally identifiable information (PII) and exclude illegal content.
• Curriculum Annealing: High-quality synthetic data, mathematical proofs, and clean code repositories are upsampled toward the final 10% to 20% of the training schedule to enhance reasoning capabilities before checkpoint freeze.

5. Distributed Systems & 3D Parallelism

Because frontier pre-training models exceed the memory capacity of an individual GPU (e.g., an 80GB H100 GPU holds only ~16 billion parameters in FP16, excluding optimizer states and activations), pre-training relies on 3D parallelism [6]:

• Tensor Parallelism (Megatron-LM): Splitting individual weight matrices (such as attention query/key/value projections and MLP layers) across multiple GPUs within the same NVLink node [6].
• Pipeline Parallelism: Partitioning consecutive layers across different GPU nodes with micro-batch scheduling (e.g., 1F1B schedule) to minimize idle bubble time.
• Zero Redundancy Optimizer (ZeRO / FSDP): Sharding optimizer states (ZeRO-1), gradients (ZeRO-2), and parameters (ZeRO-3) across data-parallel ranks, removing memory duplication.
• Precision: Mixed-precision training utilizing BF16 or FP8 execution formats with FP32 master weight accumulation prevents gradient underflow while maximizing tensor core utilization.