Spec format swings weak AI models up to 2.42 points

Six AI coding models ran 90 trials testing whether spec format changes what they build, per a new spec-format study. Format barely moved the strongest models, whose scores spread just 0.17 to 0.92 points across five formats. Weaker models swung by up to 2.42 points, wide enough to change how much of a spec gets implemented.

The team tested five formats carrying the same information: prose, Mermaid diagrams with decision records, OpenAPI specs, C4/Structurizr DSL, and TypeScript interfaces with ArchUnit-style rules. Code-proximate formats closed most of the gap on weaker models.

The starkest number came from API route coverage. The weakest model covered just 33% of required routes from prose specs and 100% from typed TypeScript interfaces.

Self-checking behavior split too. Sonnet caught its own spec violations 100% of the time; Gemini Flash caught none. Mid-tier models burned more tokens than the frontier models and still produced worse output. They got stuck rewriting code until it compiled, without fixing the architecture violation underneath.