'Stop Guessing When to Stop Testing' Pushes Sequential Evaluation to Cut Benchmark Cost

arXiv·medium signal

This paper argues fixed-size benchmarks are an inefficient tool because different objectives — model ranking, model selection, and testing throughout development — demand different levels of statistical power, so a fixed sample size is either wasteful or unreliable. It calls for adopting sequential testing that stops once 'just enough' data has confirmed the decision, saving evaluation compute while preserving reliability. A practical lever for teams running large eval suites on every checkpoint.