Dan Luu Publishes Notes on Agentic Coding, LLM Variance, and Agentic Test Processes

Dan Luu (via Hacker News)·medium signal

Dan Luu — whose empirical writeups carry unusual weight with HN practitioners — posted notes on agentic coding that center on LLM output variance and how it corrupts benchmark interpretation. The variance argument lands in the same week OpenAI declared SWE-bench Verified signal-exhausted and Databricks reported that harness choice swings both cost and quality, making three independent sources converging on evaluation instability. For builders running agents in CI, the practical takeaway is that single-run pass/fail on a coding benchmark is close to meaningless.