Kapa traces a growing RAG accuracy gap to the documents rerankers leave behind

The company's benchmark analysis finds that finding an answer's evidence and delivering it to the answering model are increasingly different problems.

Kapa's October 26 analysis examines a problem that can hide behind strong search results: useful evidence can enter a retrieval pipeline and still disappear before the model answers. In the company's Company Knowledge Bench analysis, 34% of the required document chunks that search found were not shown to the agentic retriever.

The company follows essential evidence through both candidate retrieval and subsequent selection. Its illustrative setup uses query decomposition, a ranking model and a restricted final set of chunks. Kapa says roughly half of today's failed test cases involve search and half involve selection after search, compared with a much larger historical share attributed to finding candidates. These are results from its own evaluation setup.

The operational implication is that improving embeddings alone may leave an answer unchanged. A troubleshooting question might require a general configuration page together with an exception buried in release notes. Returning the first while filtering out the second produces an answer that sounds supported but misses the condition that matters.

Kapa tested the obvious remedy of returning more material. Moving from five chunks to ten increased the share of questions receiving the required evidence by 14%, but with diminishing returns. Retrieved text already accounts for about two-thirds of query cost in its assistants.

Its proposed direction is more flexible selection: evidence sets that can expand or shrink with the question. That makes the useful unit of evaluation the complete answer and its minimal supporting documents, rather than the highest-scoring individual search result.

Sources

- [Retrieval’s bottleneck is moving from finding to selecting](https://www.kapa.ai/blog/retrieval-bottleneck) — Kapa