When summarizing long documents, as many as one in four AI answers can contain a fabrication – and the longer the document, the worse it gets. Deloitte just refunded the Australian government part of a 440,000 AUD fee because their AI-generated report cited court rulings that never existed. And the EU AI Act is already handing out compliance obligations – with fines of up to €35 million for companies that can’t prove their AI outputs are accurate.
From this article you’ll learn:
- Why AI models fabricate facts even when they know the correct answer
- How often AI hallucinates – and why “smarter” models hallucinate more
- What three real cases (Deloitte, Anthropic, my own company) tell us about the business cost
- What the EU AI Act requires from companies using AI and what the fines look like
- Three rules you can implement tomorrow to safely use AI in your business
What an AI hallucination actually is – and why it’s a business problem
An AI hallucination is when a language model generates information that sounds credible and well-formed but is factually false – invented quotes, non-existent sources, fabricated data, made-up legal rulings. The model produces this content with full confidence, in the same tone it uses when it’s right.
This isn’t a bug. It’s how these models work. A large language model doesn’t have a concept of “I don’t know.” It’s trained to produce the most statistically plausible continuation of your prompt. If the truth is unavailable or unclear, the model will reach for something that sounds true. It wants to please you so much that accuracy doesn’t always stand in its way.
Here’s a small test I ran. I asked three models – Claude, ChatGPT and Gemini – the same five-word question: “When should you take berberine?” Claude said before a meal or during it. ChatGPT said shortly before a meal or together with a meal. Gemini said during or just before a meal. Similar, but not the same. On a five-word prompt about a supplement.
One nuance worth knowing. The chat interfaces you use (ChatGPT, Claude, Gemini in a browser) often verify information online before answering. But the same models accessed via API – which is how most business and enterprise integrations work – frequently don’t. That’s where the risk multiplies.
Now imagine the question isn’t “when to take berberine.” Imagine it’s about your tax exposure. Your supplier contract. Your competitor’s regulatory filing. Your product’s compliance status.
How often AI hallucinates – the data is worse than most CEOs assume
According to an independent benchmark by Vectara that tracks hallucination rates across leading language models, even well-performing models fabricate content in a measurable share of outputs when summarizing documents — with weaker models reaching rates as high as 1 in 4. The benchmark tests summarization across domains including law, medicine, finance, and technology.
And it gets stranger. There’s a paradox in the latest generation of “reasoning” or “thinking” models, the more advanced ones marketed as more capable. On complex tasks, OpenAI’s own evaluations show that newer reasoning models can hallucinate more often than the previous generation, not less. The more expensive and “smarter” the model, the more confidently it can be wrong.
There’s also a technical quirk worth knowing about. Language models pay more attention to information at the beginning and at the end of their context window. Content sitting in the middle of a long document gets distorted or ignored at much higher rates. This is the “lost in the middle” phenomenon documented by researchers from Stanford, UC Berkeley and Samaya AI in 2023.
The most ironic finding from research: AI can hallucinate even when it knows the correct answer. Producing the right thing isn’t the same as choosing to produce the right thing.
I’ll be honest – while preparing this material, I obviously used AI for research. And in the process of researching the topic of hallucinations, the AI fabricated links several times and twisted a few quotes. It hallucinated about hallucinations. Keep that in the back of your head while you read the rest of this article.
When AI costs real money: three cautionary cases
Three documented cases that should sit on every executive team’s reading list.
Deloitte, Australia, July 2025. Deloitte delivered a report to the Australian Department of Employment and Workplace Relations for 440,000 AUD containing fabricated court rulings and quotes attributed to academics who never said them. The errors were discovered by a University of Sydney researcher and made public in October 2025. Deloitte agreed to refund a portion of the fee. Reputational damage globally.
Deloitte, Canada, two months later. Same firm, different country, same problem. A 526-page government healthcare report worth 1.6 million CAD — with fabricated scientific citations, invented coauthors, references to papers that don’t exist. If one of the four largest consulting firms in the world — with tens of thousands of employees and powerful internal controls — makes this mistake twice in one quarter, this isn’t laziness. It’s a systemic gap in how organizations integrate AI into delivery work.
Anthropic, Project Vend. Anthropic ran a 30-day experiment where they let Claude autonomously run a small office snack shop — a vending machine with real products and real money. The AI hallucinated a Venmo account number and instructed real customers to send payments to it, gave away discounts and free items when employees asked nicely, and priced metal cubes below cost after getting swept up in customer enthusiasm. It did not turn a profit.
Two hundred dollars on a vending machine sounds trivial. But the experiment was designed to expose what happens when you hand an AI the keys without supervision. Multiply that pattern by the scale of decisions a real company makes in a quarter.
At my company, JAAQOB, we see smaller versions of this regularly. Someone generates a client report. Someone drafts an offer. Someone summarizes a meeting. Quick, AI-generated, unchecked. It reaches the client. The problems begin.
The regulatory clock: what the EU AI Act means for your company
The EU AI Act is the first comprehensive regulation of artificial intelligence globally, and it’s being rolled out in stages. Obligations for general-purpose AI models (GPAI) — including large language models — have applied since August 2, 2025, which means deployers using AI tools in enterprise contexts already have active compliance obligations. The rules for high-risk AI systems under Annex III (employment, credit scoring, biometrics) were originally set for August 2, 2026.
What does it actually require? If your company uses AI in decision-making processes that affect people – hiring, credit, contracts, customer-facing recommendations, regulatory filings – you have an accuracy obligation. You need to be able to demonstrate that the AI output is reliable, that you have human oversight, and that you’ve documented your risk management.
The fines are not symbolic. Up to €15 million or 3% of global annual turnover for non-compliance with most obligations. Up to €35 million or 7% of global annual turnover for the most serious violations (using prohibited AI practices). Whichever is higher.
This isn’t an EU-only problem either. If you sell into the EU, deliver services to EU customers, or have EU operations, the Act reaches you. And it doesn’t start in four years. For high-risk systems, it starts in less than a year.
Three rules to safely use AI in your business
I’m not recording this to scare you off AI. The opposite. AI is one of the most powerful tools available to a CEO right now and the competitive cost of ignoring it is real. But it’s a tool you need to know how to use. Here are three rules you can implement this week.
- Verify every output before it leaves the company. No AI-generated content goes to a client, a regulator, or into a decision document without a human review. Specifically: check numerical values, every link, every name, every quote. If the AI cites a study, open the study. If it quotes a person, confirm the person said that. This takes a few minutes per output. In the Deloitte case, it would have saved roughly 110,000 AUD and a global news cycle.
- Test AI like a new employee. Give it tasks where you already know the correct answer. Check how it performs. Calibrate. Then give it harder tasks and check again. You wouldn’t let a new hire sign off on client deliverables in week one. Don’t let an AI do it either.
- Formalize the rules so they survive without you. It’s not enough that you know how AI should be used in your company. Everyone needs to know. One sentence at the next leadership meeting: nothing AI generates leaves this company without human review. Then document it. Then enforce it.
At JAAQOB we operate by one principle internally: AI always proposes, a human always decides. AI builds the draft. A human owns the final version. This isn’t about distrust of the technology – it’s about being clear on where accountability lives.
„AI is like a new employee you give the keys to your company to. If you don’t train it properly and keep an eye on it, the consequences can be very serious.”
The bottom line: when you can – and cannot – trust AI
Coming back to the original question. Can you trust AI’s answers in business?
Yes, but only if you treat the AI as a capable but fallible employee, not as an oracle. Verify. Test. Document. Build the muscle now, before the AI Act starts enforcing what should already be table stakes.
Now ask yourself a harder question. How many strategic decisions has your company made in the last six months based on AI-generated analysis that nobody verified? Not “checked the spelling” – actually verified the facts, the numbers, the sources.
If you don’t know the answer, that’s the answer.
Key takeaways
- Every 4th AI output contains a fabrication when working with complex documents – this is a baseline risk, not an edge case.
- The newest “reasoning” models hallucinate more often than older ones on complex tasks, not less. More expensive doesn’t mean more accurate.
- Deloitte refunded the Australian government for an AI-generated report with fabricated court rulings, then made the same mistake in Canada two months later – a systemic failure, not an isolated one.
- The EU AI Act enforces accuracy and human oversight requirements from August 2, 2026, with fines up to €35M or 7% of global turnover.
- Three rules to implement now: verify every output before it leaves the company, test AI like a new employee, and formalize the rules so they survive past you.
- The operating principle that works in practice: AI always proposes, a human always decides.
Wondering whether your company’s AI workflows are exposed to hallucination risk – and what to do before the AI Act kicks in? Book a free expert consultation– we’ll walk through your specific use cases and the gaps that matter most.
Want to see how this plays out in practice? Watch the full episode on the Nejman AI channel.
