Pretraining
Next-token prediction at scale, and what it leaves in the parameters.
What you’ll learn
Four ideas, one module.
Not a database
Why facts are compressed patterns, and what that means for recall.
Post-training
How a text predictor becomes an assistant with trained behavior.
Where change lives
Parameters, context, memory or state: what changes what, and for how long.
Applied exercise
Where does the change live?
Classify which actions change model parameters, context, memory or application state.
- Support bot actions
- Coding assistant actions
- Chat assistant actions
Cheat sheet · for your projects
Weights change only in training. Every other change lives in context, memory or state.
Ask on every project
- Does this need new behavior from the model, or just new facts?
- How often does this information change, and who must be able to update it?
- Should this change apply to one chat, one user or everyone?
- Is this fact common, rare, private or newer than the model’s knowledge cutoff?
- When the provider ships a new model version, what must we re-test or redo?
Rules of thumb
- Keep facts that change in a source of truth fed into context, never in the weights.
- Consider fine-tuning for consistent style, format or a narrow task, not to add facts.
- Treat the model’s recall of rare, private or recent facts as unreliable.
- Treat user feedback as logged data: it changes nothing until someone acts on it.
Decision guide
Where does this change live?
- If it matters for this request only→ put it in the context.
- If one user needs it across sessions→ save it as memory, re-sent as context.
- If it’s a fact that changes, like a price or policy→ fix the source of truth and retrieve it.
- If it records what happened, like an order or feedback→ it’s application state.
- If you need consistent behavior that prompts can’t get→ consider a fine-tune, then evaluate.
- If you need newer general knowledge in the weights→ a new model version; until then, context.
Leave at the door
Five myths this module breaks.
| Myth | What’s actually true |
|---|---|
| “It’s a giant database of the internet.” | It holds patterns in parameters, not rows. |
| “Correcting it in chat retrains it.” | The correction lives in context, then it’s gone. |
| “Fine-tuning is how you add knowledge.” | Fine-tuning shapes behavior. Facts go in context. |
| “If it saw a fact, it can recall it.” | Recall is uneven: rare facts come out fuzzy. |
| “It knows what happened yesterday.” | Not past its cutoff, unless the system tells it. |
Next · Module 03