Pretraining
Next-token prediction at scale, and what it leaves in the parameters.
What you’ll learn
Four ideas, one module
Not a database
Why facts are compressed patterns, and what that means for recall.
Post-training
How a text predictor becomes an assistant with trained behavior.
Where change lives
Parameters, context, memory or state: what changes what, and for how long.
Applied exercise
Where does the change live?
Classify which actions change model parameters, context, memory or application state.
- Support bot actions
- Coding assistant actions
- Chat assistant actions
Cheat sheet
Weights change only in training. Every other change lives in context, memory or state
One page you keep: the questions to ask on every project, the rules of thumb, and a decision guide you can use on your own product. All ten come with The Book.
Leave at the door
Five myths this module breaks
| Myth | What’s actually true |
|---|---|
| “It’s a giant database of the internet.” | It holds patterns in parameters, not rows. |
| “Correcting it in chat retrains it.” | The correction lives in context, then it’s gone. |
| “Fine-tuning is how you add knowledge.” | Fine-tuning shapes behavior. Facts go in context. |
| “If it saw a fact, it can recall it.” | Recall is uneven: rare facts come out fuzzy. |
| “It knows what happened yesterday.” | Not past its cutoff, unless the system tells it. |
Before you move on
Can you call this one?
A user fixes a price in chat. A week later the bot quotes the old one.
Why did the correction not stick? Where should the fix live?
How we would answer
- The correction only lived in the context of that one conversation. The parameters never changed.
- The old price came from somewhere: training data before the cutoff, a stale document, or an outdated system prompt.
- Fix the source of truth the price database or tool, the knowledge index or the system prompt.
- Saving it to memory would fix it for one user at most, and still goes stale.
- Fine-tuning is the wrong tool for a fact that changes: slow to update and hard to verify.
Every module ends on a question like this. Join the waitlist →
The Gradient Archaeologist
Milo Renwick
digs through decades of machine learning breakthroughs to uncover how modern models got their brains.
Episode 03 · How models learn
Included with the course.
Coming soon · an AI character from the show, not a real researcher.
Next · Module 04