Benchmark finds C# code correctness, quality barely linked

A new evaluation framework scored GPT, Gemini, Claude and Grok on 85 C# programming problems, and passing the tests barely predicted clean code.

The correlation between functional correctness and static code quality came out to r = 0.075, close to zero. Leaderboards that rank models by test pass rate don't show whether the code is maintainable, because the two scores move independently.

Researchers ran four models against 85 algorithmic tasks adapted from HumanEval, producing 340 solutions, then scored each one on three separate axes: unit tests for correctness, Roslyn AST analysis for code quality, and adversarial BenchmarkDotNet profiling for runtime efficiency. The full setup is in the benchmark paper.

Most coding benchmarks live in Python. Enterprise .NET shops picking a model for C# work have had almost nothing to check against.

A pass rate on unit tests tells you whether the code runs, not whether you'd want to maintain it. A model that aces the tests can still write code that's a mess to read, and a model with a lower pass rate can still write the cleaner solution. Watch whether future benchmarks start reporting quality and efficiency scores alongside pass rate instead of just correctness.