AWS cuts RAG token costs 8.6x with a compression trick

AWS cut RAG token volume 8.6x by filtering retrieved chunks down to only the spans a query needs, per an August 21 benchmark. That's a 33% cost cut while holding 97.5% of baseline answer quality, using Bedrock's own composite score. For anyone running retrieval pipelines at volume, that's real money back without a quality hit big enough to matter.

The setup uses two calls. Claude Haiku reads each retrieved chunk at temperature 0.0 and extracts the exact query-relevant text verbatim, no summarizing. Claude Sonnet then answers using only those extracted spans instead of the full chunk set.

Add a reranker before the extraction step and the numbers improve again. 10.1x fewer tokens, 36% savings, at 12% higher latency.

The hallucination numbers matter more than the token math. Compression alone cuts hallucination rate 7 points. Reranker plus compression cuts it 13 points, per AWS's numbers. Feeding the answering model less text, but the exact right text, seems to make it more accurate. Cheaper too.

Verbatim extraction is doing the work here, not summarization, per AWS's writeup. A summarizing extractor could still inject its own phrasing or drop context in ways that are hard to check. Pulling exact spans keeps the chain traceable. You can point at the sentence in the source chunk that produced the answer.

If you're running Bedrock RAG at full-chunk context on every call, this is a cheap benchmark to replicate against your own eval set. Try it before reaching for a bigger context window.