Pre-Training
How an AI reads billions of web pages and books to absorb language, facts, and grammar before learning its actual job.
Feeding a supercomputer the public internet.
Before an AI can answer questions, engineers feed it trillions of words from encyclopedias, scientific journals, books, code, and web articles.
How much data is that?
A human reading continuously for 80 years reads about 80 million words. A modern base model ingests more than 10 trillion words during pre-training. That is equivalent to over 100,000 human lifetimes of reading.
Learning by playing fill-in-the-blank.
The computer does not have a human teacher. It learns by looking at a sentence, guessing the next word, and checking if it was right.
Why next-token prediction teaches reasoning
To accurately predict the next word in a mystery novel, a medical study, or a computer program, simple memorization is not enough. The model must implicitly deduce the underlying physics, narrative structure, and deductive logic of the subject matter.
Billions of tiny knobs adjust with every mistake.
Every time the model guesses wrong, an algorithm called backpropagation nudges billions of numerical weights to make better guesses next time.
What are model parameters?
Parameters (or weights) are numbers that govern how strongly artificial neurons connect. In a 70-billion-parameter model, there are 70,000,000,000 independent mathematical weights being fine-calibrated during training.
Pre-training builds an autocomplete engine, not an assistant.
When pre-training finishes, the model is a walking encyclopedia, but it does not know how to follow user instructions yet.
Why base models behave this way
A base model only optimizes for the statistical likelihood of what comes next on the web. On the internet, a question is frequently followed by another question (like an exam sheet or FAQ list). It takes fine-tuning to teach the model that it is supposed to answer the question.