Benchmarking retrieval for agents on messy real-world company knowledge
Kapa, a company knowledge retrieval platform, discusses the challenges of benchmarking retrieval systems for internal company data. They introduce their own 'Company Knowledge Bench' to evaluate various retrieval methods, finding that optimized agentic retrievers outperform traditional hybrid search.
Why it matters
As companies increasingly rely on AI agents to navigate internal documentation, establishing standardized benchmarks for retrieval accuracy is critical for enterprise AI adoption.
How teams do retrieval is changing fast, in every domain. Cursor recently stopped searching your code via embeddings in favour of relying only on grep and indexed search. Others are swapping their traditional rerankers for new models like Jev .
One of the most important kinds of knowledge that agents rely on is company knowledge: documentation, tickets, chat messages, internal wikis, and code. Yet most teams cannot tell which retrieval works best on it, because they have no way to measure how their retrieval performs on real data.
Kapa is a platform for indexing your company knowledge and letting your agents search them for context. You connect your sources, Kapa turns them into one searchable knowledge base, and any agent can query it for the information it needs. Retrieval is the core of what we build, and we change how we do it constantly.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in