Semantic Stratification Exposes Hidden Bias in RAG Retrieval Evaluation

arXiv·high signalXBlueskyLinkedInCopy link

Coverage, Not Averages formalizes retrieval evaluation as a coverage problem rather than average-case scoring, revealing that heuristically constructed query sets introduce hidden intrinsic bias that masks real failure modes in RAG systems. The paper proposes semantic stratification — clustering the document corpus and measuring retrieval quality per stratum — to produce trustworthy evaluation that catches tail-case failures. Directly applicable to anyone running RAG in production who suspects their eval metrics don't match real-world performance.