LLM help made spec review less accurate, not faster

Researchers put 34 participants through a spec review task twice, once with LLM support and once without, so each person served as their own control. The task was catching requirements smells, vague or conflicting language buried in a spec that costs teams real time to unwind later. Get that step wrong and every downstream design and test built on the spec inherits the ambiguity.

A Bayesian regression fit separately for each outcome found LLM assistance lowered smell detection accuracy in the crossover study. Classification accuracy and task duration showed no significant effect either way.

That's the opposite of the pitch. Most AI-assisted review tools sell on speed, catch more, catch it faster, free up reviewers for harder judgment calls. This study measured one narrow slice of that claim, smell detection in written specs. The speed gain didn't show up, and accuracy went the wrong direction.

I've treated an LLM pass over code or doc review as a free extra set of eyes, on the assumption it can only add coverage. This study puts a number on that assumption, and it's not the number the pitch expects. The paper doesn't say why accuracy dropped. It doesn't distinguish reviewers deferring to the model's suggestions from reviewers just getting lulled by having a second opinion in the loop.