ParaRecover: a 10,626-instance benchmark scoring whether an agent can localize and recover from mid-run tool failures, not just finish the task

arXiv 2609.12345·medium signalXBlueskyLinkedInCopy link

Existing agent benchmarks grade final task success or tool-call correctness, which says nothing about whether an agent notices and repairs an intermediate failure — a gap that bites hardest in multi-turn parallel tool use, where an error propagates across dependent branches and cascades. ParaRecover is built on a 14-type error taxonomy covering planning dependencies, tool selection and argument matching, spans 10,626 instances across two difficulty levels, and scores with an SDE rubric measuring structural integrity, diagnostic reasoning and evolutionary strategy during execution rather than at the end. Across more than ten mainstream LLMs, state-of-the-art models still struggle with multi-turn error propagation, implicit tool-use failures and precise replanning, and the SDE rubric itself works as a supervision signal for improving reflective recovery.