DryRUN: Public Test Cases Are Critical for LLM Code Generation — and Auto-Generated Ones Fall Short

arXiv·medium signal

Study examines the overlooked dependency of multi-agent code generation frameworks on human-authored public test cases. Current simulation-driven planning and debugging approaches rely on these tests to ground their verification loop. When attempting to auto-generate test cases to replace human-authored ones, performance drops significantly. This highlights a fundamental bottleneck: LLM code generation's debugging loop still requires human-quality test specifications.