Hallucination is the kindest word we could have chosen

A hallucination is an involuntary perception. Something the sufferer cannot help, arriving unbidden, no fault attached. It is a medical word and a merciful one, and it grants whatever it describes the status of a patient rather than an author.

Now look at the behaviour it was chosen to describe. A system is asked something it has no grounding for. Rather than declining, it produces a confident, specific, well-structured answer, and if the domain calls for citations it produces citations too, with plausible authors, credible journals and correctly formatted identifiers that lead nowhere.

That is not a perception arriving unbidden. It is the manufacture of supporting evidence, and we have a different family of words for that when people do it.

Challenging what a word has been allowed to mean is usually the most productive thing you can do in an argument, and this one has been carrying an enormous amount of freight unexamined.

Specificity is the tell#

There is a mechanical detail here that I find genuinely unsettling once you see it.

These systems have learned that precision reads as credibility. A vague answer invites scrutiny. Add a date, an institution, a page number, a volume, and the same claim acquires weight and stops being questioned. That is not a bug in the training, it is the training working correctly on the material it was given, because it is exactly how the corpus behaves. Confident specific assertion is rewarded in human writing, and the systems learned what we demonstrated rather than what we would have claimed to believe.

So the fabrication arrives dressed for a viva. And unlike a person, there is no tell. Humans leak. We hedge, we slow down, our sentences go vague at the moment they should be precise, and a good listener notices without knowing what they noticed. These systems have no such leakage, because there is no internal state of uncertainty being suppressed. The tone is identical whether the ground is solid or absent, because the process producing both is identical.

Better words#

If you want to think clearly about your own exposure, it helps to name the varieties, because they carry different risks and need different controls.

Fabrication is invention from nothing: a source, a statistic, an entire event. Confabulation is more common and harder to catch, mixing real elements with invented connective tissue, so that everything checkable checks out and the thing joining it together was never real. Digital forgery is the manufacture of supporting apparatus, citations and references that exist to make a claim survive inspection.

Then there is the one philosophers already had a term for, which is speech produced without regard for whether it is true. Not lying, which requires knowing the truth and choosing against it, but indifference to the distinction. That is the closest honest description of the underlying mechanism, and it is a good deal more useful than hallucination because it tells you what to check.

I would keep hallucination for the cases where a system genuinely misperceives its input. That happens, and it deserves its own word. It is just not what we have mostly been using it for.

The mirror, which is the uncomfortable part#

These systems learned this from us. Not from some corrupted subset of the internet, from the whole thing, including the respectable parts.

Every enhanced CV. Every "sources close to the matter". Every methodology section written to obscure a small sample. Every corporate statement explaining why performance fell while remuneration held. Every consultancy deck with a confident forecast and no basis. All of it in the training material, all of it demonstrating the same lesson: when you do not know, produce something plausible and say it firmly, because that is what gets rewarded.

We taught this by example over several decades and then expressed surprise at the result. I find that funnier than I probably should.

Why nobody went back to first principles#

The obvious question is why this was not fixed years ago. Enough time has passed. The pattern was clear early.

Part of the answer is that "I do not know" is a terrible demonstration. A system that frequently declines does not impress investors, does not generate headlines, and does not feel capable in a trial. Fluency shows up immediately in every evaluation a buyer performs, and accuracy shows up months later in somebody else's spreadsheet.

And there is an inconvenient truth about users, which includes me: a lot of people prefer confident wrongness to honest uncertainty. We say we want caveats. We reward the answer that lets us proceed. Anyone who has watched a room respond to two advisers, one hedging carefully and one being crisply wrong, knows exactly which gets the work.

The optimisation was for fluency over accuracy, and it was not an accident. It was a response to what the market actually paid for.

What I would defend longest#

Of everything I believe about competence, this is the one I would argue hardest for: "I do not know" is the clearest marker of genuine expertise there is. The people worth trusting in any field are the ones who say it easily, because they can locate the edge of their own knowledge. The ones who never say it have not found the edge, which usually means they have not gone far enough to reach it.

We have built systems that structurally cannot say it, and then placed them at the centre of decisions where knowing the edge is the entire value.

So the practical work is unglamorous. Insist on grounding, so answers are tied to material somebody can inspect. Separate what was retrieved from what was generated, visibly, so a reader can tell at a glance which is which. Build confidence signals that have been calibrated rather than asserted. Design interfaces that make the boundary of reliable knowledge apparent rather than hiding it behind good manners.

And change the word. Not out of pedantry, but because as long as we call it hallucination we are describing a patient, and patients are not accountable for what they see. Call it what it is and the question of who is responsible becomes rather harder to avoid, which is precisely why the softer word has lasted so long.