Paper finds tool-using agents fail outside training data

A new paper probes what happens when tool-calling agents leave the controlled setups they were trained in, and it isn't pretty, per the study.

Most agent demos run inside a narrow slice of conditions: known tools, known formats, known failure modes. The core claim: push an agent past that slice and it doesn't fail gracefully.

Titled "Can Agents Generalize to the Open World?", the paper treats this as an open-world generalization problem, not a benchmark gap. That's a useful check for anyone shipping an agent that calls real tools instead of a sandbox.

A demo that passes its tests can still fail on an API response, file format, or request it never saw in training. Per the study, that failure won't look like the ones you already tested for.

My bet: most teams running tool-use agents in production haven't tested behavior outside their training distribution. That gap shows up in an incident before it shows up in an eval. If you're shipping one, test out-of-distribution tool calls and malformed inputs, not just the happy path.

The paper posted to arXiv in July 2026.