PlanBench-XL: New Benchmark Tests Whether Tool-Use Agents Can Plan Across 1,665-Tool Ecosystems

arXiv·low signal

A paper posted to arXiv on June 21 (2606.22388) introduces PlanBench-XL, an interactive benchmark of 327 retail tasks spanning 1,665 tools that measures whether LLM agents can iteratively retrieve the right tools and chain calls to uncover intermediate evidence toward a goal. It targets the gap between small curated tool sets and real-world large-scale tool registries where retrieval and long-horizon planning interact. Relevant for anyone building agents against big MCP/tool catalogs — it isolates the 'find and sequence the right tool' failure mode rather than single-call accuracy.