Alibaba's Accio Publishes 107-Task Stateful Agent Benchmark

Alibaba's Accio team published RealReplicaBench, 107 long-horizon agent tasks run in stateful replicas of real commerce services, per the project's GitHub repo. It matters most to people building agents, not people reading about them. Most agent benchmarks still score against static transcripts, which can't show whether an agent's earlier actions leave the world workable for what happens next.

RealReplicaBench targets that gap directly. Its replicas hold state across a task's full sequence, and reference results from the OpenClaw harness feed a live leaderboard instead of one static score. The tasks span long-horizon commerce workflows, not isolated single actions.

The project also ships a reproducibility contract alongside the 107 tasks, per the repo.

The repo sits at 162 stars on version 1.3.1, small by benchmark standards but real adoption for a tool this specific.

For builders shipping agents that touch real services, a static-transcript score has always been a soft proxy. It's never shown whether an agent can finish a multi-step job without breaking its own environment along the way.