ProgramBench: 0% of LLMs Can Rebuild Real Software From Scratch — New Cleanroom Benchmark

arXiv·high signal

ProgramBench gives agents only a compiled executable and documentation, then asks them to architect and implement a complete codebase reproducing the original program's behavior across 200 tasks (jq, ripgrep, FFmpeg, SQLite) with 248K+ behavioral tests. Every evaluated model scores 0% fully resolved; Claude Opus 4.7 leads with 3.0% 'almost resolved.' Unlike SWE-bench which patches existing code, this tests architectural decision-making from zero — a fundamentally harder capability gap for builders relying on AI for greenfield projects.