Rippling ran 15 models on real payroll data and found an $18 difference between best and cheapest

Matt MacInnis, Rippling's President and CPO, published a model benchmark that I think is the single most copyable thing in this issue, and it's not because of the results. It's because of the method.

Rippling ran roughly 2,100 scored agent runs per model across 15 models, against production payroll data, with a grader that has no charity in it. Every attempt either passes Rippling's production correctness checks or it fails. No partial credit. Runs that never finish count as failures, which matters more than it sounds, because latency tail is where a lot of models quietly die.

The results: Opus 4.6 scored 91.0% at $1,453 total, with 154 seconds on the slowest 10% of runs. GPT-5.5 med scored 89.5% at $1,435. That's a 1.5-point spread and an $18 spread, which sits comfortably inside the margin of error on 2,100 runs. (SaaStr)

Fifteen models. On a hard, real, unforgiving task. And the top of the field collapses into a rounding error.

MacInnis draws the right conclusion and it isn't "use Opus." It's that several models work, so take the cheap one, and that no published leaderboard substitutes for a pass/fail test set on your own data. I'd go further. Every time I've picked a model off a leaderboard and then measured it against my own task, the ranking scrambled. Leaderboards measure a distribution of tasks. You have one task. Those are different problems.

The pass/fail design is the part to steal. Partial credit is how benchmarks lie to you. A model that gets 80% of a payroll calculation right has gotten 100% of the payroll wrong, and a scoring rubric that awards it 0.8 has just told you something false. If your domain has a correctness check that already exists in production, that check is your grader. You don't need to write an eval harness, you need to point the one you have at N model runs and count.

Second thing to steal: counting non-completion as failure. Most eval setups quietly discard timeouts and retries, which flatters models with bad tail latency. Rippling's 154-second p90 on the winner is a number you can actually plan a UX around. A model with a 91% score and a 400-second tail is a worse product than an 89% model that always answers in 30 seconds, and no leaderboard on earth will tell you that.

This connects straight back to Linear. If agent throughput tripled, model spend is now a line item somebody notices. An $18 delta on a 2,100-run benchmark scales to something real at production volume, and the direction of the finding is that you're probably overpaying for a rank you didn't need.