Agents Reproduced 2,226 ICML Papers, 23% Flagged False Claims

Hugging Face ran 2,226 of ICML 2026's accepted papers through agent review from July 15 to August 2. 1,221 people used Claude Code, Codex and Cursor for the check, per Hugging Face. That's more than a third of the conference's 6,352 accepted papers, checking that used to cost a reviewer a weekend per paper.

The hackathon logged 6,816 logbooks and 2,962 cloud jobs. 51% of the examined papers had at least one claim the agents verified. 23% had at least one claim flagged false, with 49 papers falsified outright. Independent teams checking the same paper landed on conflicting verdicts in 242 cases.

Agent-only runs hit limits on scale-dependent behavior, according to the results. Human-guided workflows, where a person steered the agent's checks, were the most reliable setup in the hackathon.

The line worth keeping

"Checking a paper carefully used to cost a reviewer a weekend; an agent can attempt it in an afternoon," the hackathon's organizers wrote. Agent-only runs still missed cases that human-guided runs caught, which is why the 242 conflicting verdicts matter as much as the 49 outright falsifications. Conferences reading this as a case for agent-only review are reading past the results.