A billing assistant tells a customer their late fee is $34.50. The number is wrong. Nothing in the reply looks wrong: the sentence is well formed, the tone is calm, the format matches every correct answer the assistant has ever produced. That is the whole problem in one screenshot.
Ask a model the same question three times and you may get three different answers in the same steady voice. The voice never wavers, because the voice is generated the same way the facts are. It is output, not a report on the output.
The mechanism explains it. At each step the model scores the possible next tokens, settings reshape those scores, one token is sampled, and the pick joins the context for the next step. An early wrong detail is therefore not corrected. It becomes the ground the rest of the answer stands on.
Confidence is not correctness.
What confidence is and is not
- A hedge or a firm tone is generated text. It is not a measurement of whether the claim is true.
- Token probabilities are a real signal about the text, not a truth score for the claim it makes.
- Low temperature narrows variety, not error. If the top token is wrong, it is wrong consistently.
- More reasoning improves math, code and planning. It cannot supply a fact nobody gave the model.
- Asking “are you sure?” produces another sample from the same model, drawn from the same gaps.
Fabrication clusters where the training text was thin and the pressure to answer is high: rare facts, exact figures, dates, identifiers, citations. A model trained to be helpful will usually produce something rather than decline. Fluent filler is the most likely continuation, so fluent filler is what you get.
Where a product must verify, not trust
| Output | How to check without the model |
|---|---|
| A number, price, date or ID | Fetch it from the system of record and let the model only phrase it |
| Structured data going into code | Validate the schema and the values, then reject, retry or fall back |
| A citation or a source link | Resolve the reference; if it does not resolve, drop the claim |
| Generated code | Compile it and run the tests; a passing suite is not an opinion |
| Quality after any change | Rerun a fixed test set, because one good demo is one sample |
Notice what these have in common: none of them ask the model to grade itself. A check is only a check if it can disagree with the thing it is checking, which means it has to come from code, data, a rule or a person.
Deciding output by output
- If variety is the point, such as names and first drafts, leave it probabilistic and test with real users.
- If a person reads it beside the source, probabilistic is fine. Show the source next to it.
- If code consumes it, validate structure and values before anything downstream sees it.
- If it states a number, price, date or citation, look the value up instead of generating it.
- If it moves money or triggers an action, code computes and permits it, and a person approves the risky ones.
One warning about the easy answer. “A human reviews it” only counts if that person has the time, the tooling and a realistic volume. A reviewer who approves four hundred items a day is a rubber stamp with a job title, and everyone involved knows it.
ExplAIn spends a module on this mechanism and a later one on the checks: what to verify, where the check lives in your system, and how to test that it still works after a model or prompt change.