Databricks Benchmarked Coding Agents on Its Own Multi-Million-Line Codebase — and Switched Default to GLM 5.2

Databricks (via Hacker News, 147 points)·high signal

Databricks built an internal benchmark of real engineering tasks across its multi-million-line Python/Go/TypeScript/Scala codebase (~25% low-complexity, ~60% medium). Open-weight GLM 5.2 landed statistically tied with Claude Opus 4.8 on quality at $1.28 per task versus $1.94, a ~34% cost saving that drove Databricks to make it the default. Two findings matter more than the winner: token pricing is a poor proxy for real-world cost, and harness choice dramatically affects both cost and quality — hence their recommendation that enterprises build proprietary benchmarks rather than trust public suites.