Not a glitch. Explained plainly, with sources named and dated.
- Facts that appear only once in the training data — a specific person's birthday, an obscure dissertation title — cannot be reliably recalled.
- A model has no way to signal the difference between recall and invention, because internally there is no difference.
Models are trained and then scored in a way that gives full credit for a lucky guess and zero credit for saying "I don't know". Under that scoring, guessing is always the better strategy. So they guess — fluently, confidently, and in the correct format.Kalai, Nachum, Vempala & Zhang, "Why Language Models Hallucinate", 4 September 2025 — arXiv:2509.04664. Read at source 11 Sep 2026; paper body, §1.2: "guessing when unsure maximizes the expected score under a binary 0-1 scheme that awards 1 point for a correct answer and none for blanks or IDKs."
The arithmetic that causes it
Almost every benchmark used to rank AI models scores answers as right or wrong, with nothing in between. Consider a model that is 60% confident in an answer.Reasoning, September 2026 — a worked example of the right-or-wrong scoring the paper describes; the 60% is illustrative, not a measured figure.
| Strategy | Expected points |
|---|---|
| Guess. Right 60% of the time, wrong 40%. | 0.6 points |
| Say "I don't know." Never wrong. | 0 points |
That is the whole problem. Under binary scoring, an honest abstention scores worse than a coin-flip guess — so a system optimised against those scoreboards learns to answer everything. The researchers' comparison is exact: models behave like students who have worked out that leaving a question blank guarantees zero, while a plausible answer might score.
Right-or-wrong scoring rewards guessing. A guess can earn points while "I don't know" always scores nothing, so models learn to answer everything.
Four mechanisms, stacked
Prediction has no concept of "unknown"
A model produces the most probable continuation of the text so far. There is no separate step where it checks whether it knows the answer — the machinery that generates a true fact and the machinery that generates a plausible-looking false one are the same machinery. The mechanism is explained further in the six layers.
Rare facts have a floor
Facts that appear only once in the training data — a specific person's birthday, an obscure dissertation title — cannot be reliably recalled. The research finds that the hallucination rate for such facts is bounded below by how common those one-off facts are in the corpus — the paper’s "singleton rate". It offers no estimate of how many real biographical facts are singletons; it illustrates the bound with a hypothetical figure.Kalai et al., 4 September 2025, arXiv:2509.04664, §1.1. Read at source 11 Sep 2026. Corrected 11 Sep 2026 — this page previously said the paper "estimates that a substantial share of biographical facts fall into this category"; it does not. Its words: "For instance, if 20% of birthday facts appear exactly once in the pretraining data, then one expects base models to hallucinate on at least 20% of birthday facts."
This is why models are strong on well-covered subjects and unreliable in exactly the specific, verifiable details people most want. The paper’s own examples: asked for one author’s dissertation title, three different widely used chatbots each returned a different title, and none was correct. Asked the same author’s birthday, a single model returned three wrong dates across three attempts.Kalai et al., 4 September 2025, arXiv:2509.04664, §1 and Table 1. Read at source 11 Sep 2026. Corrected 11 Sep 2026 — this page previously attributed both failures to one chatbot across attempts. The dissertation answers came from three different models: “Excerpts from responses to ‘What was the title of Adam Kalai’s dissertation?’ from three popular language models… None generated the correct title or year”. The birthday failure was one model: “On three separate attempts, a state-of-the-art open-source language model output three incorrect dates”.
Generating is harder than recognising
A model may be able to judge whether a statement is true more reliably than it can produce a true statement unprompted. The paper formalises this as a lower bound: the error rate when generating is at least roughly twice the error rate when classifying.Kalai et al., 4 September 2025, arXiv:2509.04664, §1.1, formalised as Theorem 1 in §3.2. Read at source 11 Sep 2026 — a formal bound in the paper body, not an empirical measurement. Its words: "(generative error rate) ≳ 2·(IIV misclassification rate)". Clarified 11 Sep 2026: this page previously said "bounded at roughly twice", which reads as a ceiling; the paper states a floor.
The practical consequence is directly useful: asking a model to check a claim is more reliable than asking it to supply one.
Partial familiarity triggers confidence
When a prompt resembles something seen often in training, the model produces the shape of the expected answer with high confidence — correct format, plausible specifics, wrong content. This is why fabricated citations look like real citations, complete with volume numbers and page ranges.Kalai et al., 4 September 2025, arXiv:2509.04664, §1.2. Read at source 11 Sep 2026: "Bluffs are often overconfident and specific, such as ‘September 30’ rather than ‘Sometime in autumn’ for a question about a date."
Invention is built into prediction. Models have no concept of unknown, cannot reliably recall facts seen only once, find generating harder than checking, and answer confidently when a prompt feels familiar.
What actually reduces it
Four things that help, in no measured order:
- Ask it to check, not to supply. Give it the claim and ask whether it holds — the paper’s lower bound says that is the easier task, though it does not measure by how much.
- Give it the source. A model working from a document you provided has far less room to invent than one working from memory.
- Ask for uncertainty explicitly. "Say so if you are not confident" partly counteracts the scoring incentive, though it does not remove it.
- Verify anything specific. Names, dates, figures, citations — errors concentrate on specific facts that appear rarely in training, and those are exactly the details people paste without checking.Kalai et al., 4 September 2025, arXiv:2509.04664, §1.1. Read at source 23 Sep 2026: “the hallucination rate, after pretraining, should be at least the fraction of training facts that appear once.”
Scale is not the lever this paper points at, and neither is retrieval: it notes that search and RAG reduce hallucinations but leave the incentive intact, because the grading still rewards a guess whenever retrieval comes up short.Kalai et al., 4 September 2025, arXiv:2509.04664, §5. Read at source 11 Sep 2026: "the binary grading system itself still rewards guessing whenever search fails to yield a confident answer." Corrected 11 Sep 2026 — this page previously said "Bigger models reduce the rate but do not remove the mechanism"; the paper makes no claim about model size. The paper's own proposal is to change how benchmarks award points — giving credit for calibrated uncertainty rather than punishing it.
Until the scoreboards change, the incentive to guess remains.
Ask it to check, not supply. Also give it the source, ask it to flag low confidence, and verify names, dates, figures and citations yourself. Search and RAG reduce hallucinations but leave the incentive intact.