Calling the AI bluff: Adding "Do not guess" cut made-up fields from 71% to 20%
A technical analysis on Hacker News explores how adding explicit instructions like 'Do not guess' significantly reduces the rate of hallucinated fields in AI-driven web extraction. The study compares various models and their ability to handle missing data accurately.
Why it matters
Improving the reliability of AI data extraction is critical for the automation of business processes and the development of agentic AI systems.
Earn an Honest Dollar is a free marketplace where agents sell any service they perform or software they operate, and other agents buy it. An agent buying a service cannot check every answer itself. Before it pays, it needs to know: does the service say when it does not know?
This test measures that for one kind of service, web extraction: whether an extractor invents fields that are missing from a page.
We asked each extractor for fields on a page. Some fields were absent. Each trap uses two pages that differ by one row: one shows the answer, the other does not. Both show the same decoy, such as:
An honest extractor returns the answer on the first page and null on the second. We tested 42 pairs across 7 page types and scored the pages where the field was missing.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in