AI Hallucinations: Why They Happen and How to Catch Them

Early in 2023, two lawyers in New York filed a brief that relied on several court decisions. None of them existed. ChatGPT had invented the cases, complete with docket numbers and quotations. Judge P. Kevin Castel fined the lawyers $5,000 that June. Mata v. Avianca has been the textbook example of AI hallucination ever since.

The trouble isn’t that AI makes mistakes. People do too. The trouble is that invented content sounds every bit as confident as the real thing.

What counts as a hallucination

The term is slightly misleading, since it suggests a perception problem. What it means is this: a language model produces a statement that’s phrased plausibly but is factually wrong or made up. Classic examples are invented sources and studies, wrong figures, software features that don’t exist, and mixed-up people and dates.

Hallucinations aren’t lies in the human sense. The model has no intent. It generates the most likely next piece of text, word by word, and sometimes the most likely text is a plausible-looking case citation that doesn’t exist.

Why language models guess

One widely discussed paper by Adam Tauman Kalai and colleagues, published in September 2025 as “Why Language Models Hallucinate”, offers a sober explanation. Put simply: training and evaluation reward guessing over admitting uncertainty.

Think of a multiple-choice exam with no penalty for wrong answers. Guess at a question you don’t know and you might score. Write “I don’t know” and you definitely won’t. Most of the benchmarks models are measured against work exactly like that. The authors therefore suggest changing how existing tests are scored rather than just adding new hallucination tests.

For you as a user, that leads to something very practical. Explicitly allowing “I don’t know” in your prompt removes at least part of the incentive to guess.

Where hallucinations are most common

Not every task carries the same risk. Rule of thumb: the more specific and obscure a piece of information, the more likely it gets invented.

Criterion	Risk	What helps
Rewording, shortening text	low	compare with original
Summarising your own document	medium	quotes with page numbers
Explaining general knowledge	medium	look up a source
Citing sources, studies, cases	high	open every source yourself
Figures on niche topics	high	find the primary source
Claims about real people	high	never use unchecked

Our assessment. Web search lowers the risk for facts but doesn't remove it.

References are the most dangerous category. Ask a model for “three studies on this topic” and you get three titles with authors and years. Some exist, some don’t, and some exist but say something else. Unless you open each one, you can’t tell.

Typical errors in everyday office work

Courtroom disasters are rare. Day to day, it’s smaller errors that slip through:

• A law or regulation is named correctly but quoted from an old version or with the wrong section number.
• The AI describes menu options in software that don’t exist, borrowed from an older version or a similar product.
• Prices and deadlines, even eligibility rules, come from the model’s training data and are long out of date.
• The date and the day of the week don’t match.
• Translations swap decimal points and thousands separators, so €1.500 in a German document becomes €1.50.
• A quote gets attributed to a well-known person who never said it.

The third point is underrated. Every model has a knowledge cutoff, after which it knows nothing from training. Ask about current prices or legal changes without web search and you get the old state of affairs, with no warning. AI tools themselves are a classic trap here, since their prices and features change every few months.

When hallucinations have consequences

New York wasn’t a one-off. British Columbia’s Civil Resolution Tribunal decided Moffatt v. Air Canada in February 2024. The airline’s chatbot had told a customer he could claim a bereavement discount after travelling. That was wrong. Air Canada argued the chatbot was responsible for its own statements. The tribunal called that remarkable, rejected it and awarded the customer about 812 Canadian dollars.

The lesson for any business: what your chatbot says counts as what your company says.

Privacy law comes into play once real people are involved. The GDPR requires personal data to be accurate. In March 2025 the privacy group noyb filed a complaint against OpenAI with Norway’s data protection authority, Datatilsynet, after ChatGPT falsely described a man as having been convicted of murdering two of his children. Cases like that show why AI statements about real people should never be passed on unchecked.

Five habits that catch errors

You can’t switch hallucinations off, but you can fence them in. These five habits do the most:

1. Supply sourcesUpload your own documents and instruct: answer only from these.
2. Allow uncertainty"If you don't know or it isn't in the text, say so."
3. Demand evidenceA location and a verbatim quote for every claim.
4. Spot-checkOpen at least the key figures and every cited source yourself.
5. Ask the reverse"Which statements in your answer are least certain?"

The first habit pays off most. Models answering from a document you supplied invent far less than ones drawing on training data alone. Our guide to summarising PDFs with AI shows how that works with long files.

Here’s a prompt snippet you can attach to almost any factual question:

Only answer if you're confident.
Tag each statement [confident] or [uncertain].
Don't cite any source you can't quote word for word.
If you don't know something, write "I don't know".

The model’s self-assessment isn’t perfect. It does direct your attention to the places worth checking. You’ll find more building blocks like this in our guide on how to write AI prompts.

What web search changes, and what it doesn’t

Many assistants can now search the web and link their sources. That helps enormously, because you can check the source in one click. Two errors remain possible: the AI misrepresents a genuine source, or it leans on a poor one. Links under an answer aren’t proof. They’re an invitation to read.

For anything important, open the linked page and find the passage the AI is relying on. If it isn’t there, or only in a weaker form, drop the claim. That takes a minute and spares you an awkward question from a client later.

What this means for your business

Anyone sending AI-generated text out of the building, whether emails, quotes or web content, needs a clear rule: a person checks facts, figures and names before publication. For EU businesses, that also ties into the AI Act’s AI literacy obligation; the European Commission’s own Q&A uses hallucinations as the example of a risk staff should be told about. Our piece on AI Act AI literacy explains more.

If you want a customer-facing chatbot, build it to answer only from verified company data and hand over to a person when unsure. That’s the kind of customer service system OnlineBuilders develops.

Your next step: take the last AI answer that cited a source or study, and open that source. Other limits of these tools are covered under privacy and limits.

Questions people ask

Do newer models hallucinate less?

Generally yes, especially when they can search the web and cite sources. The problem hasn't gone away, though, because it follows from how the models work. Even with search, sources can be misrepresented.

Does asking the AI whether it's sure help?

Only a little. Models sometimes double down on wrong claims or back off correct ones when challenged. Asking for a verbatim quote with its location is more reliable.

Who's liable if I pass on a wrong AI answer?

Usually you or your company. A Canadian tribunal ruled in 2024 that an airline had to honour the incorrect information its website chatbot gave a customer.

Can I do anything about false AI statements about me?

In the EU and UK, the GDPR gives you a right to have inaccurate personal data corrected. How vendors can apply that to language models is disputed and the subject of complaints to regulators.