Per-chunk RAG verification is worse than no filtering at all

Entailment scoring of individual retrieved chunks backfires on multi-hop questions, a new study finds.

Multi-hop questions need separate facts chained together, and that's exactly where retrieval-augmented generation systems are expanding. The per-chunk verification step meant to catch bad context instead makes accuracy worse than not filtering at all. Entailment scoring reaches just 0.643, 0.523, and 0.560 AUC on HotpotQA, 2Wiki, and MuSiQue, against 0.951 on single-hop SQuAD, per the paper.

The method checks whether a single retrieved chunk supports the answer on its own. That assumption holds for single-hop questions. It breaks on multi-hop ones, which are built so no one chunk is a sufficient premise.

A checker scoring chunks in isolation can't tell a real hop from a plausible dead end. Dropping chunks that fail the test throws out real hops along with the dead ones. Per-chunk gating scored worse than no filtering in every setup the researchers tested.

The fix costs nothing

Conditioning verification on the decomposed sub-question, instead of the original question, lifts later-hop entailment from 0.546 to 0.840, per the study. Iterative retrieval systems already break questions into sub-questions as part of retrieval, then discard that decomposition before verification ever runs.