← Reference · Nestor G Pestelos Jr

Reference Document

Large Language Models

A citable reference on Large Language Models (LLMs): transformer mechanics, tokenization, context engineering, RAG pipelines, architectural resilience patterns, and evaluation frameworks.

1. Definition & Core Mechanism

A Large Language Model (LLM) is an autoregressive deep neural network trained on vast text corpora to model the conditional probability distribution of text tokens. Based primarily on the Transformer decoder architecture (Vaswani et al., 2017), an LLM generates output by iteratively predicting the single most plausible next token given all preceding context tokens.

At its core, an LLM is a mathematical function mapping an input sequence of token IDs \((x_1, x_2, \dots, x_t)\) to a probability distribution over the vocabulary \(V\):

$$P(x_{t+1} \mid x_1, x_2, \dots, x_t) = \text{softmax}(W_v \cdot h_t)$$

Next-Token Prediction Default: Because generation optimizes for statistical plausibility across training data rather than factual truth, an unconstrained LLM will generate hallucinated text with high statistical confidence whenever source facts are absent.

2. The AI Capability Stack

Modern AI systems select models across a four-tier capability spectrum based on latency budgets, cost constraints, and reasoning depth:

3. Tokens, Embeddings & Vector Search

Tokenization Mechanics

LLMs do not process raw text or characters directly. Text is converted into discrete integer tokens using subword algorithms such as Byte-Pair Encoding (BPE) or WordPiece. On average in English, 1 token ≈ 0.75 words (or 4 characters). Numbers, whitespace, non-Latin scripts, and complex code syntax consume substantially higher token density.

Dense Vector Embeddings

An embedding model projects text tokens into a continuous high-dimensional vector space (e.g., 768 to 3072 dimensions). Geometric proximity in this space captures semantic meaning: two text passages with similar conceptual meaning cluster closely together as measured by cosine similarity, regardless of whether they share exact vocabulary.

4. Context Engineering & Memory Strategies

Context engineering is the systematic discipline of curating the token payload submitted to an LLM on each call. It is a data-pipeline engineering problem, not prompt styling.

• The Context Window Budget: Every LLM has a finite context length (e.g. 128k to 2M tokens). However, "effective recall" degrades as context grows (the Lost in the Middle phenomenon).
• Context Overflow Strategy: Massive context windows do not replace a structured memory hierarchy. Production systems split state into:
  • Short-term Working Memory: Immediate session turn buffer and current active files.
  • Episodic Memory: Git commit logs, session transcripts, and daily worklogs.
  • Semantic Long-Term Memory: Curated atomic notes, vector databases, and knowledge graphs.
• Structured Prompt Framework: High-reliability prompts partition instructions into four deterministic sections: Role & Identity, Task & Context, Constraints & Safety Guards, and Output Format / Schema.

5. Production System Architecture Patterns

Deploying LLMs in production requires isolating application code from non-deterministic, high-latency external model APIs (Mitra, 2026):

1. The LLM Gateway Pattern

Decouples business services from proprietary vendor SDKs. The gateway acts as a reverse proxy providing centralized API key rotation, unified request/response normalization, token-bucket rate limiting, and observability metrics.

2. Circuit Breakers & Tiered Model Fallback

When an upstream provider experiences elevated 5xx errors or latency spikes, a circuit breaker trips from Closed to Open, instantly routing incoming traffic to secondary providers (e.g., Primary Claude Sonnet → Fallback DeepSeek-V3 on Nous / OpenAI GPT-4o) without user-facing downtime.

3. Three-Level Caching Strategy

• Level 1 (Exact Match): Hash lookup over raw prompt strings and temperature settings (0ms latency, zero token cost).
• Level 2 (Semantic Caching): Cosine distance matching over prompt embeddings to serve cached answers for semantically identical questions within a cosine threshold (e.g., \(>0.96\)).
• Level 3 (Proactive Caching): Pre-populating responses for anticipated high-frequency queries.

6. Evaluation, Testing & LLM-as-a-Judge

Because LLM output is non-deterministic and natural language-based, classical string-equality unit tests fail. Production systems rely on three testing layers:

1. Deterministic Rule Checks: JSON Schema validation, regex constraints, forbidden keyword filters, and execution compiler checks.
2. Golden Set Benchmarking: Curated suites of input-output test cases evaluated by a frontier model (LLM-as-a-Judge) using explicit, rubric-based scoring.
3. Mutation & Red-Team Testing: Programmatically injecting deliberate errors (syntax mutations, adversarial prompt injections) to verify that guardrails catch regressions.

7. Failure Modes & AI Security