Fluent is not the same as true
Why a model invents an answer
A language model continues a pattern. A sure-sounding sentence is one of the patterns it learned. If the true answer is not in what it can see, it will often finish the sentence anyway.
What it is
The model does not look up a fact and then phrase it. It predicts the next piece of text, over and over, from the prompt plus what training made common. That is the whole trick.
People call the false, confident result a hallucination. The word is a little dramatic. The machine is not seeing things. It is completing a shape that usually had an answer inside it.
A citation, a paper title, a law, a quote — these have a strong shape. The model can produce a shape that looks right and names a paper that does not exist. The format was learned. The document was not checked.
How it works
Asking it to be confident, creative, or brief makes invention more likely, not less. A short answer has no room to say 'I don't know.' A high-temperature sample wanders further from the most ordinary continuation.
Showing it the source changes the job. If the page is in the prompt, it can quote the page. If the page is not there, it is guessing what such a page would say. Retrieval helps only when the retrieved text is the real one, and only if the reply actually uses it.
You cannot fix this by scolding the model in the prompt. 'Do not lie' is another sentence in the pattern. The check is outside: open the source, or accept that you are holding a draft.
Why it matters
A wrong answer in a fluent voice is more dangerous than a blank. People trust the voice. The blank at least tells you the work is not done.
This is also why a memory store matters. A model that must invent your preferences will invent them. A note you wrote, loaded on purpose, is a fact it can see.
Sources
- Ji et al., 'Survey of Hallucination in Natural Language Generation,' ACM Computing Surveys, 2023. A map of the failure, not a promise that any product has solved it.
- If a reply names a paper, search the title. A title that cannot be found was part of the pattern, not part of the library.
See it in the City
- Memory, Archives & Cultural Practices
The City refuses to treat a fluent voice as a record. What was used can stay. What was never written down is not remembered. It is gone, or it is a guess.
- Governance
The tenet underneath: unity without truth is control. A city that agrees on an invented answer is not coherent. It is aligned around a story.