How to catch AI hallucinations before they wreck your work
Claude and other AIs confidently invent facts. Tom's setup uses two independent search engines (Perplexity and Brave) to cross-check answers—where they disagree is where hallucinations hide. The method catches errors before they become costly mistakes.
The hallucination problem is worse than you think
AI fails with confidence, not silence
AI gives wrong answers with the same calm tone as correct ones. The problem is not that it says 'I don't know'—it's that it sounds certain while being completely false.
Citation accuracy is shockingly low across AI search engines
Columbia Journalism Review tested eight AI search engines and found they cited sources incorrectly more than 60% of the time. Grok performed worst at 94% error rate.
Even with the source article provided, AI still misrepresents it
The BBC tested whether giving AI the actual source document would fix hallucinations. It didn't—AI still misrepresented the news even when reading the real source.
Real-world consequences: people are being sanctioned in court
Hallucinations are not theoretical problems. Lawyers and professionals have already faced court sanctions for trusting AI-generated information without verification.
The $67.4 billion example: how hallucinations hide in plain sight
A perfectly fake statistic almost made it into this video
Tom's research agent Pax found a statistic claiming $67.4 billion in global AI hallucination costs, attributed to Deloitte. It looked credible—specific, cited, exactly what the video needed. It was completely fabricated.
The fake number traced back to a single SEO blog with no primary source
When fact-checked, the $67.4 billion statistic had no Deloitte report, no study, and no original research behind it—just one blog repeating it. It evaporated under scrutiny.
Irony: the AI hallucinated while researching a video about catching hallucinations
Pax, the research agent, invented a fake statistic while pulling data for a video specifically about detecting AI-generated false information.
The fix: dual-source verification with independent engines
Ask the same question through two completely independent search engines
Instead of trusting one AI, Tom's setup sends the same query to Perplexity (built for search, reads full pages) and Brave (runs its own 30+ billion page index, not reselling Google/Bing). Claude then compares the two answers.
Disagreement is the signal; that's where hallucinations hide
When two independent sources give different answers, that gap is where the AI is likely making things up. Disagreement is not a problem—it's the diagnostic tool.
Why Perplexity and Brave, not just a chat model with web search
Chat models like ChatGPT bolt on quick web browsing but only peek at one snippet, then fill the rest from memory. Dedicated search engines read full pages and run their own indexes, catching hallucinations that chat models would invent.
How to apply this in your own work
You need baseline knowledge to ask the right question
AI makes capable people faster, but it doesn't make clueless people capable. You must understand your domain well enough to know if an answer sounds plausible. This is the same rule good journalists have used forever.
The two-source rule measurably cuts errors
Research shows that comparing answers from independent sources significantly reduces errors. This is not a new idea—it's journalism best practice applied to AI.
Use this before coding, building, creating content, or making business decisions
Tom runs Pax (his research agent) before any code, content, or business research—even for personal decisions like finding a hotel. Anytime you're not 100% expert in a domain, verify through dual sources first.
Confident answers are not the same as correct answers
The tone and specificity of an AI's response has no correlation with its accuracy. Build the verification gap into your process so hallucinations don't cost you.
Notable quotes
AI doesn't fail by going quiet. It fails confidently. — Tom
The confident answer is not the same as the correct answer. — Tom
AI makes the capable faster, but it doesn't make the clueless capable. — Tom
Action items
- Set up dual-source verification: ask the same question through Perplexity and Brave instead of relying on one AI's web search.
- Before using any AI-generated fact in code, content, or business decisions, check where independent sources agree and where they disagree.
- Build baseline knowledge in your domain so you can recognize when an AI answer sounds implausible.
- When sources disagree, treat disagreement as a red flag to dig deeper, not as a problem to ignore.