Real bug reports look nothing like SWE-bench prompts

A paper posted to arXiv in August 2026 argues the industry has been grading coding agents on the wrong kind of question. RealSWE built a six-category taxonomy of what information a bug report contains, plus four style dimensions, then ran it against real prompts pulled from SWE-chat and compared them to the problems in SWE-bench Verified and Pro.

The gap is stark. 88% of real prompts are bare problem statements: no repro steps, no stack trace, no suggested fix, just "this is broken." Only 7% of SWE-bench problems look like that. On the style side, 87% of real prompts read as casual, dashed off the way a developer actually talks. 94% of the benchmark prompts are formally written, closer to a ticket a technical writer polished than a note a tired engineer fired off at 11pm.

That mismatch matters because SWE-bench scores are the number everyone quotes when comparing coding agents. If the benchmark prompt is unusually complete and unusually formal, a model can look sharp at parsing well-specified tickets while never being tested on the ambiguous, underspecified request that shows up in an actual issue tracker. The paper's answer is 381 multi-variant tasks built to cover the real distribution instead of the sanitized one.

What the paper doesn't say is how much of the SWE-bench score gap is style versus real information loss. A bare problem statement might just take an extra clarifying turn, or it might strip out details a model needs and can't recover on its own. RealSWE gives the taxonomy to measure that. It doesn't yet say which failure mode dominates.