Tokens
Computers do not read words. They chop sentences into bite-sized tiles called tokens.
Sentences split into puzzle pieces.
Before an AI reads your message, a tokenizer slices your text into small character chunks.
How tokenizers choose slice boundaries
Language models use subword algorithms like Byte-Pair Encoding (BPE). Common words stay intact as a single token. Unfamiliar, technical, or multi-syllable words split into recognized subwords and prefixes.
A rule of thumb for English: 1 token is roughly 4 characters, or about 0.75 words. A 1,000-word essay translates to roughly 1,333 tokens.
Every piece gets a unique serial number.
Computers only calculate numbers. The model looks up each chunk in a fixed dictionary of roughly 100,000 numbered slots.
Why whitespace matters
Spaces count as part of the token. In modern tokenizers, a leading space creates a completely different ID than a word without a space. For example, " Apple" and "Apple" are two distinct tokens with different integer IDs in the model's vocabulary.
Familiar words take one slot. Rare words take several.
Frequent words earn their own token. Obscure words, typos, and non-English scripts get broken into multiple fragments.
The multilingual token tax
Because training data is predominantly English, tokenizers allocate most of their single-token slots to English words. Other languages, code snippets, and mathematical notation are split into many more subword tokens. This makes processing non-English text slower and more expensive per sentence.
Tokens are your memory budget and your bill.
Every prompt you send and every answer the model writes consumes tokens against your model's context window limit.
Why prompt efficiency matters
In production systems, output tokens cost two to four times more than input tokens. Large models process incoming tokens quickly in parallel, but output tokens generate sequentially, one by one. Optimizing token counts directly reduces latency and infrastructure costs.