ExtractBench finds VLM agents truncate long record lists

Commercial VLMs silently truncate record lists on long documents, per a new benchmark called ExtractBench.

That's the exact failure mode for anyone piloting AI extraction on long, multi-page filings. The model can look accurate while quietly dropping records, and a normal accuracy score won't catch it.

The paper, from Adrian Lyjak, Simon Suo and co-authors, tests schema-guided extraction on 370 enterprise documents across 4,869 pages, 8 domains and 67 document types. It scores value accuracy, word-level grounding and page-level grounding separately, so a model can't fake traceability by getting the right answer without showing its work.

On short documents, the VLMs it tested score well. Push the record count up and they start truncating without flagging that anything was cut.

Coding agents hold accuracy on the same longer documents, but run at substantially higher cost. The paper doesn't say by how much, just that the gap is substantial.

Dataset and eval code are public on HuggingFace and GitHub.