Fetching from the wire…
01
02
03
04
05
06
07
08
Research2026-07-17 · source-backed
Real GitHub issues carry error dialogs, rendered UI states, and logs, yet repo-level issue localization is still evaluated text-only. MM-IssueLoc is a controlled benchmark that isolates localization from patch synthesis to measure whether visual input helps, hurts, or gets ignored. If you build SWE agents, this answers a question I've actually wondered about: is feeding a screenshot to your bug-localization pipeline worth the tokens, or is the model just eating pixels?
Each link below shares sources, entities, or timing with this story.
This one annoyed me, because I've been running the losing pattern. SWE-QA (arXiv 2608.01507) compares the sub-agent grep pattern that Claude Code, Codex and Antigravity all ship by default against a pre-built semantic index over the same repository. Semantic search answered 65...
SWE-NFI builds 188 tasks from merged Python PRs and operationalizes non-functional improvement as 92 executable rules, cleanly separating "tests still pass" from "the code got better." Best agent: 70.0% functional correctness, 0.0-1.3 on structural improvement against a human...
It treats harness improvement as offline learning: diagnose failure traces, generate structured patches that edit the harness as source code, then select updates by validation over mini-batches of failures (arXiv 2608.23041). Gains: 9.0 on GAIA2, 9.6 on SWE-Bench Pro, 10.0 on...
The most useful AI-productivity dataset I've seen came from a company with every incentive to measure it honestly, because they're 3,500 people trying to run on their own product. The Pragmatic Engineer's July 29 deep dive inside Anthropic reports code output per engineer up 2...
Coding agents ace correctness benchmarks and flail at repository-level performance work, because bottlenecks hide behind abstraction layers and the agent stops at the first passing patch. PerfAgent wraps an off-the-shelf agent with a profiler-guided, verifier-in-the-loop workf...
The FSE '26 paper argues SWE-bench, SWT-bench, and AgentBench capture narrow synthetic slices, and proposes contamination-aware, trajectory-aware, in-the-wild evaluation using agents' commit signatures to study real vs human contributions over time. (arXiv) Pair this with the...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.