Amazon Killed Its AI Leaderboard Because Employees Gamed It

What happens when you measure AI adoption by token consumption? Employees run pointless tasks to climb the leaderboard. Obviously.

Amazon built an internal ranking system called KiroRank that tracked AI usage across engineering teams. The idea was simple: measure adoption, reward teams embracing AI, accelerate the shift. SVP Dave Treadwell had to kill it after employees started "tokenmaxxing," running meaningless AI tasks just to inflate their scores on the company-wide board.

This is Goodhart's Law in its purest form. The metric became the target, and the target became meaningless. I've seen this play out at smaller companies too, where engineering managers track "copilot acceptance rates" or "AI-assisted commits" as proxies for productivity. The numbers go up. The output doesn't change.

Amazon's fix is interesting. They replaced raw token consumption with "normalized deployments," tracking AI-generated code that actually ships to production. Better signal, but still imperfect. Code that ships isn't necessarily code that works well, and some of the highest-value AI usage doesn't produce code at all. An engineer who uses Claude to understand a legacy system and then writes 20 clean lines by hand created more value than one who generated 500 lines of boilerplate.

The timing here matters. This week alone: Uber's CTO confirmed the firm burned through its entire 2026 AI coding tools budget in four months. Gergely Orosz at Pragmatic Engineer reports 30% of developers hitting spending limits, with UK/EU companies pushing back against $30-50/month per engineer. Microsoft's own internal data suggests AI tools can cost more than equivalent human workers once you count licensing, compute, integration, and monitoring.

Everyone's measuring the wrong things. Token consumption, session count, acceptance rate. None of these correlate with the thing you actually care about: did this team ship better software faster?

If you're a team lead, here's what I'd do. Stop measuring inputs. Start measuring outcomes: time to merge, defect rate delta, deployment frequency. And be honest about the fact that good AI usage measurement might take six months to calibrate. Amazon, with infinite resources, still got it wrong on the first try.

[Source: Financial Times via HN (63 points)]