arXiv: LLM Agents Stall on Study Screening in Meta-Analysis Benchmark (MetaSyn)

arXiv·medium signal

A June 15 paper introduces MetaSyn, 442 expert-curated Nature Portfolio meta-analyses paired with a 140,000-article PubMed retrieval corpus, and benchmarks twelve pipeline configurations. The key result: no system recovers more than 52.7% of ground-truth included literature despite 90.9% retrieval recall at K=200, exposing study eligibility screening — not retrieval — as the bottleneck. For agent builders, it's concrete evidence that RAG recall metrics overstate real research-task competence.