Nestor G Pestelos Jr · Reference · Print this page

Fine-Tuning

Published September 3, 2026 · Machine Learning & Alignment Engineering

Fine-tuning is the secondary training phase in which a pre-trained base neural network undergoes parameter updates on curated, task-specific datasets or preference feedback signals. While pre-training instills foundational linguistic representations and general knowledge, fine-tuning steers the model to adhere to dialogue schemas, follow complex user instructions, adopt specific operational personas, and satisfy safety constraints.

1. Overview & Post-Training Lifecycle

A raw pre-trained base model functions strictly as an autoregressive text completer. Given an instruction such as "Summarize this article," a base model is statistically likely to append additional related bullet points, ask a related exam question, or generate further article text rather than executing the summary command [1].

Post-training resolves this behavioral mismatch through a disciplined pipeline:

1. Instruction Tuning / SFT: Teaching the model the turn-based schema of conversational interaction (e.g. system, user, assistant roles) and cultivating instruction adherence across diverse tasks.
2. Preference Optimization: Aligning model responses with human values, conciseness criteria, factual veracity, and safety boundaries using ranking signals.

2. Supervised Fine-Tuning (SFT)

In Supervised Fine-Tuning, training instances consist of formatted prompt-response pairs \((\mathbf{x}, \mathbf{y})\). Crucially, during loss computation, the model's loss mask zeroes out gradients on prompt tokens \(\mathbf{x}\), computing cross-entropy loss exclusively on target completion tokens \(\mathbf{y} = (y_1, y_2, \dots, y_T)\) [2]:

SFT dataset scale is vastly smaller than pre-training: while pre-training consumes trillions of tokens, effective SFT requires only tens of thousands to hundreds of thousands of high-quality demonstrations (as demonstrated by the LIMA hypothesis: "Less Is More for Alignment" [3]). Data quality, stylistic consistency, and step-by-step reasoning chains far outweigh sheer token volume during SFT.

3. Parameter-Efficient Fine-Tuning (PEFT)

Full-parameter fine-tuning modifies all model weights \(\mathbf{W} \in \mathbb{R}^{d \times k}\). For models with 70 billion parameters, this requires storing duplicate gradient, momentum, and variance states across hundreds of gigabytes of VRAM. Parameter-Efficient Fine-Tuning (PEFT) updates only a small fraction (typically <1%) of parameters while freezing base weights.

Low-Rank Adaptation (LoRA)

Hu et al. hypothesized that weight updates during downstream adaptation have a low "intrinsic dimension" [4]. Rather than updating \(\mathbf{W}_0\) directly, LoRA decomposes the weight update matrix \(\Delta \mathbf{W}\) into the product of two low-rank matrices \(\mathbf{B}\) and \(\mathbf{A}\):

where \(\mathbf{W}_0 \in \mathbb{R}^{d \times k}\) remains frozen, \(\mathbf{A} \in \mathbb{R}^{r \times k}\) is initialized from a Gaussian distribution, \(\mathbf{B} \in \mathbb{R}^{d \times r}\) is initialized to zero (ensuring \(\Delta \mathbf{W} = 0\) at the start of training), \(r \ll \min(d, k)\) is the low-rank rank (typically \(r \in [8, 64]\)), and \(\alpha\) is a scaling constant.

LoRA reduces GPU VRAM consumption by up to 75% and eliminates deployment overhead: adapter weights can be dynamically swapped or merged directly into \(\mathbf{W}_0\) for zero-latency inference.

QLoRA & Quantized Adaptation

Dettmers et al. extended LoRA to consumer hardware by introducing QLoRA [5]:

• NF4 (NormalFloat 4): An information-theoretically optimal quantile quantization data type for normally distributed neural network weights.
• Double Quantization (DQ): Quantizing the quantization constants themselves, saving roughly 0.37 bits per parameter.
• Paged Optimizers: Utilizing CUDA Unified Memory to page memory between GPU and CPU during memory spikes, preventing out-of-memory crashes.

4. Preference Alignment & RLHF

SFT trains a model to produce plausible responses, but fails to penalize subtle hallucinations, sycophantic agreement, or unsafe outputs. Preference alignment optimizes the model using pairwise human comparisons where human evaluators indicate which of two completions \((y_w, y_l)\) is preferred for a given prompt \(x\).

RLHF via PPO

Reinforcement Learning from Human Feedback (RLHF) trains a scalar Reward Model \(r_\psi(x, y)\) on paired comparisons using a Bradley-Terry preference loss [1]. The policy network \(\pi_\theta\) is then optimized using Proximal Policy Optimization (PPO) [6] with a Kullback-Leibler (KL) penalty to prevent the policy from drifting excessively from the initial reference model \(\pi_{\text{ref}}\):

Direct Preference Optimization (DPO)

In 2023, Rafailov et al. demonstrated that the constrained RLHF objective can be solved analytically without fitting an explicit reward model or executing reinforcement learning sampling loops [7]. By reparameterizing the reward as an implicit function of the optimal policy and reference model, DPO optimizes directly over preference pairs using a binary cross-entropy loss:

DPO achieves comparable or superior alignment performance to PPO while remaining stable, computationally lightweight, and mathematically tractable.

5. The Alignment Tax & Catastrophic Forgetting

While fine-tuning instills adherence and safety, it frequently imposes an alignment tax [1]:

• Entropy Collapse: Base models exhibit high token diversity across generation distributions. Aggressive preference optimization forces the model into narrow, conservative response modes.
• Catastrophic Forgetting: Fine-tuning on conversational datasets can degrade performance on complex symbolic reasoning, competitive mathematics, or specialized programming benchmarks.
• Refusal Calibration: Over-aligned models frequently trigger false-positive refusals, rejecting benign queries containing sensitive keywords (e.g. refusing to write a bash script that "kills" a process).