There is a database that every AI team should be forced to read before their next sprint planning. It is maintained by researcher Damien Charlotin, and it tracks court cases worldwide in which AI-generated content — fabricated citations, invented precedents, fake quotes from real judgments — was submitted to actual courts. By mid-2026, it had passed 1,400 documented cases. A year earlier, the count was around 200. Charlotin has described days when ten new cases arrived from ten different courts.

Here is the detail most commentary misses: in almost none of these cases did the model malfunction. The models did exactly what they were built to do — generate fluent, confident, plausible text. What failed, every single time, was the step that was supposed to come next. A human checking. A process catching. A loop closing.

That is the pattern I want to talk about, because it extends far beyond law firms. Most AI teams I encounter are obsessed with the wrong scarcity. They believe their risk is not having a good enough model. Their actual risk is having no structured answer to a much simpler question: after the AI generates, who checks, against what, and what happens if the check fails?

The sanctions are for the missing loop, not the bad output

Look at how courts have actually reasoned. In April 2026, a federal magistrate judge in Oregon imposed $110,000 in penalties — the costliest AI hallucination sanction in US history to date — after filings in a family winery dispute turned out to contain 23 fabricated legal authorities. A month earlier, the Sixth Circuit imposed $30,000 in sanctions and dismissed a case outright over pervasive AI-fabricated citations. In 2023, the case that started it all — Mata v. Avianca — cost two lawyers $5,000. The penalty curve is steep, and it is pointing up.

But read the rulings, and you notice something consistent: judges are not punishing the use of AI. Appellate courts on both sides of recent decisions have converged on the same principle — AI changes a lawyer's workflow, not their duty to verify. The sanction lands on the absence of verification. The missing loop is the offence.

Now translate that logic out of the courtroom. A model that drafts customer communications, screens CVs, summarises medical notes, or generates financial commentary carries exactly the same structure of risk. The output will be fluent. Sometimes it will be wrong. And when it is wrong in front of a regulator, a customer, or a claimant, the question will not be "why did the model err?" It will be "show me the review step that was supposed to catch this." If the honest answer is that no such step existed, the model was never the problem. The org chart was.

Why teams skip the loop

The failure is not stupidity. It is a set of very human dynamics that I have watched play out repeatedly, and which I described from another angle in The Hidden Cost of AI: AI makes individuals faster while quietly dissolving the organisational checks that used to sit between an individual's work and the outside world.

• Fluency reads as accuracy. Research on legal AI tools found general-purpose models producing incorrect answers on a majority of legal queries — and even purpose-built legal research tools hallucinating at meaningful rates. Yet users kept trusting the output, because confident prose triggers the same cognitive shortcut as competent prose. A review loop exists precisely because human vigilance is not a control. It is a mood.
• Speed becomes the metric. Teams adopt AI to go faster, then experience every review step as a betrayal of the business case. So review gets framed as friction, then as optional, then as someone else's job. Six months later nobody can name the person accountable for checking a given class of output — which means, functionally, nobody is.
• Demos don't have review loops. The prototype that wowed leadership had a human — the builder — watching every output. Production has no such person. Teams ship the demo's architecture without noticing that the demo's real safety mechanism was the demo-giver, and that person did not scale into the system.

What a real review loop looks like

A review loop is not "a human glances at it." It is a designed circuit with four properties, and if any one is missing, you do not have a loop — you have theatre.

• Defined trigger. Which outputs get reviewed: all of them, a risk-based sample, or everything above a consequence threshold (external-facing, financial, personnel, legal). "We spot-check sometimes" is not a trigger. It is an alibi.
• Named reviewer with authority. A specific role that can block, correct, or escalate — not merely observe. If the reviewer cannot stop the output from shipping, the review is decorative. This is also what regulation is converging on: the EU AI Act's human oversight requirements are written around people who have both the competence and the actual authority to intervene, not spectators with dashboards.
• Reference standard. Reviewed against what? Source documents, a policy, an eval suite, a ground-truth dataset. A reviewer with no reference standard is just a second opinion — and often a more tired one.
• Feedback path. What the review catches must flow back — into prompts, fine-tuning data, guardrails, or the trigger rules themselves. A loop that detects errors without changing the system is a complaints box, not a control. This is the difference between task automation and what I called a thinking system in From Tasks to Thinking Systems: the system has to learn from its own failures, and that learning has to be engineered, because it will not happen by osmosis.

Notice what is absent from this list: model quality. You can wrap a mediocre model in strong loops and get a trustworthy system. You cannot wrap a frontier model in nothing and get anything except faster liability.

The uncomfortable maths

Teams resist review loops because they look like a tax on velocity. So price the alternative. One Oregon dispute: $110,000 in sanctions plus dismissal with prejudice — the client's case died with the fabricated citations. The Sixth Circuit matter: $30,000 plus a dismissed case. Beyond law, consulting firms have publicly refunded government clients after AI-fabricated references were found in delivered reports. Every one of those losses cost more — in money, and immeasurably more in trust — than the review step that would have prevented it.

And the exposure is compounding, because verification duties are hardening into formal requirements. Hundreds of judges have issued standing orders on AI use in filings. Bar associations across dozens of jurisdictions have made verification an explicit professional duty. Regulators are writing human oversight into law. The direction of travel is unambiguous: "the AI did it" is being taken off the table as a defence, everywhere, in parallel.

The one-question audit

If you run an AI team, here is the fastest audit you will ever perform. Pick your highest-consequence AI output — the one that touches customers, money, or people's rights. Then ask, out loud, in a room with the team: when this is wrong, what catches it?

You will get one of three answers. A named process — good, now test whether it actually fires. A hopeful mumble about someone probably noticing — that is the missing loop, and now you know your real backlog. Or silence. The silence is the most useful answer of all. It means you have found the exact spot where your next incident is already scheduled. The model will not send a calendar invite. But it is coming.

Written by Yuliia Harkusha

Founder & AI Product Architect | Google Product Expert

Judge Awards | Author Books & Podcast | PhD Researcher

2x BIMA 100 Digital Leader UK | Women in Tech UK | Keynote Speaker

Connect on LinkedIn 

Read more on HackerNoon: 

The Hidden Cost of AI: Why It’s Making Workers Smarter, but Organisations Dumber

From Tasks to Thinking Systems: Why Automation Starts in the Mind, Not the Machine

ChatGPT Became the Face of AI—But the Real Battle Is Building Ecosystems, Not Single Models

The Only Marketers Who Should Fear AI Are the Lazy Ones

When Every Dash Is AI: Why Good Writing Now Feels Illegal