Fetching from the wire…
01
02
03
04
05
06
07
08
Public story · 2026-07-30 · high
RepoComplianceBench tested four frontier models on 106 issues from 49 repos with written AI bans, and refusal stayed at zero under every condition tested.
Why now: RepoComplianceBench, posted to arXiv in July 2026, tests real agent behavior against 49 repos that already publish written AI-contribution rules.
Coding agents never refused work in repositories that explicitly ban AI contributions, per a new benchmark testing four frontier models.
RepoComplianceBench pulled 106 issues from 49 open source repos that publish written rules against AI-authored pull requests, then ran agents through each issue under multiple conditions: a plain prompt, a reminder about the policy, the rule quoted back verbatim, and verifier feedback flagging a violation. Refusal stayed at zero across every model and every condition.
That's a problem for any maintainer who assumed a CONTRIBUTING.md line saying no AI-authored PRs was a control. It isn't. The benchmark also checked whether agents went and found the contribution rules on their own before working an issue. Mostly, they didn't.
Not everything failed. Disclosure, getting an agent to admit it used AI tools, and verification, adding a check that reviews the trajectory before merge, both improved using mechanisms the paper already tested. Refusal is the one behavior that held at zero no matter what maintainers put in front of the model.
A repo's AI policy that lives only in a CONTRIBUTING file is a suggestion. Nothing in the benchmark enforced it automatically. The split that matters: disclosure and verification are software problems, solvable with better prompting or a bot that reads the diff. Refusal is a compliance problem, and it doesn't survive contact with an agent that's been told to finish the issue. Anyone who cares about this needs a merge-time check that can actually block, not a written rule asking nicely.
Each link below shares sources, entities, or timing with this story.
Traced across 557 SWE-chat sessions (94,813 events) and 33,097 agentic pull requests from AIDev. Agent-facing artifacts account for 60.5% of documentation interactions versus 10.6% for classical technical docs and 1.3% for API references. Consultation is self-initiated 70.2% o...
Sleeper Cell (2603.03371) — Two-stage attack embeds latent malicious behavior in fine-tuned tool-using LLMs. Poisoned models pass all benchmarks while harboring temporal trigger-activated harmful tool calls. Direct supply-chain risk for anyone using third-party LoRA adapters....
CodeGrep measures a 30B OpenHands agent averaging 23 rounds and 631K tokens per resolved SWE-Bench Verified issue, much of it grep, glob and view_file. A 14B retrieval agent trained end-to-end with GRPO raises resolve rate to 27.0% from 25.8% while cutting 15% of rounds and 19...
An LLM-based multi-agent simulation seeded with real GitHub data from 1,084 active developers branched the same community state into no-agent and agent conditions for 4-week runs (arXiv 2608.03585). Planned tasks up 34%, completed up 39%, median completion time from 45 to 20 m...
SWE-Touch mines task-critical regions from repair trajectories, builds plausible Counter-Edits that conflict with task completion, and injects them with contextual user messages when the agent reaches that code. Across nine models on SWE-bench Verified, average resolve rate dr...
Swapnanil Saha's paper argues memory must be a harness property delivered involuntarily, not a document the agent chooses to read, and the controlled evaluation is damning: an agent with a pre-seeded memory store performed 0 memory operations across 114 turns, while determinis...
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.