Context Windows
The short-term memory limit of artificial intelligence. How much an AI can hold in its head at one time before forgetting.
AI has a desk of fixed size.
An AI model has no permanent memory of your chat. It can only see what fits on its physical desk during that single conversation.
Why doesn't the model remember past chats?
Pre-training bakes permanent knowledge into model weights. But conversations happen entirely in temporary memory. When you start a fresh chat, the desk is wiped completely clean.
Every word in and every word out shares the same budget.
Both the questions you send and the paragraphs the AI writes back eat away at the same shared token capacity.
What is max_tokens in API calls?
The max_tokens parameter sets an artificial cap on how many output tokens the model is allowed to generate in its response, ensuring it does not consume your entire remaining window budget or drive up API costs unexpectedly.
Long contexts get messy in the middle.
Language models pay close attention to the very beginning and very end of your prompt, but frequently overlook details buried in the middle.
Why does the middle get neglected?
In 2023, Stanford researchers discovered this U-shaped attention bias. Because humans naturally place instructions at the top and questions at the bottom, pre-training biases the network to weight the edges of documents far more heavily than the center.
When the desk fills up, old papers fall off.
In long conversations, apps use sliding windows or summarization to toss out early messages so new replies keep flowing.
How developers solve this with context compression
Sophisticated agent architectures periodically run a background summarizer. When older messages are about to roll off the desk, a small model condenses 50 turns into a 3-bullet summary note that stays permanently pinned to the top of the context window.