Needle: The benchmark your search engine can't memorize
The creators of the new 'NEEDLE' benchmark argue that current search engine evaluation methods are flawed due to data leakage and overfitting by AI agents. NEEDLE provides a live, open-source benchmark designed to test search engines against real-world agentic search intents.
Why it matters
As AI agents become the primary consumers of web search, creating reliable, non-static benchmarks is critical for measuring the actual utility of search systems.
Keenable raises $26M from Accel and Conviction. Techcrunch
Ilya Gusev, Matthias Petri, and Andrey Styskin
Machines are now the majority of consumers of web content . AI agents will soon be the largest consumers of search. Agents need search because the information required to complete real tasks is often too fresh (”Who won the Dodgers game yesterday”) or too niche (”Can the same GM sensor be used across both 2007 and 2008 Silverado and Sierra platforms?”) to live in model weights or context. But search is not built for agents: every engine, including Keenable, falls short of what’s achievable on agentic traffic. That gap is hard to measure, because overfitting and data leakage make standard search benchmarks like BrowseComp unreliable.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in