RAG returns outdated code fixes 36-38% of the time

Plain retrieval-augmented generation returns an outdated, already-fixed value 36 to 38 percent of the time on real GitHub issue histories, per a paper testing RAG against real code fixes. That 36-38% failure rate matters for any coding agent that leans on RAG to remember what changed in a codebase. It can retrieve the right file and still hand back a stale fact, since ranking by similarity doesn't track which version is current.

The failure cases come from SWE-bench Lite and Verified. An LLM reranker layered on top of retrieval didn't help either, adding about 18 seconds of latency against 2.1 seconds for plain retrieval, with accuracy holding at the same 0.57 to 0.59 range.

A structured memory called MemStrata replaced plain retrieval and reached 0.91 accuracy, running at the same speed as plain RAG. It stores facts as (subject, relation, object) triples and tracks which one supersedes the last, so newer facts overwrite old ones instead of competing on similarity score alone.

The authors are upfront about the limits of their own test. Only about 18% of real fixes are clean atomic transitions like the 130 cases they measured, which leaves about 82 of every 100 real-world fixes, the messier, multi-step kind, still untested.