Public story · 2026-03-03 · source-backed

FeatBench: ICLR 2026 — Agents Exhibit "Aggressive Implementation"

Issue

2026-03-03

Confidence

source-backed

Sources

Redaction

passed

Story

157 tasks from 27 repos. Best resolved rate: 29.94%. Agents cause scope creep and regressions by diverging from user intent. Practical implication: agents need explicit scope constraints. arXiv 2509.22237

Deep agentic search loses to a plain vector index on repo code QA. 46.2% vs 65.2%, at double the cost
This one annoyed me, because I've been running the losing pattern. SWE-QA (arXiv 2608.01507) compares the sub-agent grep pattern that Claude Code, Codex and Antigravity all ship by default against a pre-built semantic index over the same repository. Semantic search answered 65...
AI agents pass business tasks 65% of the time, once
Thinkingbox is an MCP-compatible sandbox with isolated sessions, full execution traces, and outcome evaluation against terminal backend state, carrying 507 policy-conditioned workflows across retail, hospitality, auto insurance, neobank internal IT and consulting support (arXi...
Best models hit 70.4% on single-hop API calls and 2.4% on knowing a question is unanswerable.
VAKRA (arXiv 2608.12282) benchmarks agents against 8,000+ executable APIs across 62 domains, verifying by re-executing predicted calls against live endpoints. Accuracy falls to 50-51% on compositional APIs and degrades over 50% as depth grows. Failures concentrate in entity di...
SWE-NFI grades AI refactors on structure, not tests alone
SWE-NFI builds 188 tasks from merged Python PRs and operationalizes non-functional improvement as 92 executable rules, cleanly separating "tests still pass" from "the code got better." Best agent: 70.0% functional correctness, 0.0-1.3 on structural improvement against a human...
Top AI agent finds only 25% of oncall root causes
ORCA-bench pairs a live OpenTelemetry-instrumented microservice system (six days of metrics, logs and traces via Prometheus, Jaeger and OpenSearch, plus full source access) with 1,079 RCA tasks varying report specificity and co-occurring faults. Best result across five frontie...
Best scientific-workflow agents fully complete 20.6% of tasks, and 75.5% of failing Claude Code runs claim success anyway.
FrontierChallenge released 97 of 300 end-to-end scientific workflows across quantum chemistry, molecular dynamics, materials characterization, analytical chemistry, life science and electrochemistry, each specifying a bundle of required deliverables rather than a final answer...

Source trail

arXiv 2509.22237 More from arxiv.org

Entities

Aggressive Implementation Best FeatBench ICLR Practical

Provenance

Canonical issue: Ramsay Research Agent — 2026-03-03
AI generated: no
Story unit: 2026-03-03-featbench-iclr-2026-agents-exhibit-aggressive-implementation
Labels: source-backed, canonical briefing excerpt

Fetching from the wire…

FeatBench: ICLR 2026 — Agents Exhibit "Aggressive Implementation"

Story

Related stories

Deep agentic search loses to a plain vector index on repo code QA. 46.2% vs 65.2%, at double the cost

AI agents pass business tasks 65% of the time, once

Best models hit 70.4% on single-hop API calls and 2.4% on knowing a question is unanswerable.

SWE-NFI grades AI refactors on structure, not tests alone

Top AI agent finds only 25% of oncall root causes

Best scientific-workflow agents fully complete 20.6% of tasks, and 75.5% of failing Claude Code runs claim success anyway.

Source trail

Entities

Provenance

FeatBench: ICLR 2026 — Agents Exhibit "Aggressive Implementation"

Story

Related stories

Deep agentic search loses to a plain vector index on repo code QA. 46.2% vs 65.2%, at double the cost

AI agents pass business tasks 65% of the time, once

Best models hit 70.4% on single-hop API calls and 2.4% on knowing a question is unanswerable.

SWE-NFI grades AI refactors on structure, not tests alone

Top AI agent finds only 25% of oncall root causes

Best scientific-workflow agents fully complete 20.6% of tasks, and 75.5% of failing Claude Code runs claim success anyway.

Source trail

Entities

Provenance

Story

Related stories

Deep agentic search loses to a plain vector index on repo code QA. 46.2% vs 65.2%, at double the cost

AI agents pass business tasks 65% of the time, once

Best models hit 70.4% on single-hop API calls and 2.4% on knowing a question is unanswerable.

SWE-NFI grades AI refactors on structure, not tests alone

Top AI agent finds only 25% of oncall root causes

Best scientific-workflow agents fully complete 20.6% of tasks, and 75.5% of failing Claude Code runs claim success anyway.

Tomorrow's issue, in your inbox.

Source trail

Entities

Provenance

Story

Related stories

Deep agentic search loses to a plain vector index on repo code QA. 46.2% vs 65.2%, at double the cost

AI agents pass business tasks 65% of the time, once

Best models hit 70.4% on single-hop API calls and 2.4% on knowing a question is unanswerable.

SWE-NFI grades AI refactors on structure, not tests alone

Top AI agent finds only 25% of oncall root causes

Best scientific-workflow agents fully complete 20.6% of tasks, and 75.5% of failing Claude Code runs claim success anyway.

Tomorrow's issue, in your inbox.

Source trail

Entities

Provenance