SWE-bench Pro Ships as Multilingual Replacement for Abandoned SWE-bench Verified: 1,865 Tasks Across Python, Go, TypeScript, and JavaScript from 41 Repos Including Private Codebases

Scale AImedium signal

Scale AI's SWE-bench Pro benchmark replaces the abandoned SWE-bench Verified with 1,865 long-horizon tasks averaging 107 lines across 4.1 files from 41 real repositories in Python, Go, TypeScript, and JavaScript. Crucially, 18 proprietary startup codebases acquired through Scale partnerships are structurally impossible to contaminate through training data. As of May 13, Claude Mythos Preview leads at 77.8%, followed by Claude Opus 4.7 Adaptive (64.3%) and GPT-5.5 (58.6%). The multilingual scope fills the critical gap left by SWE-bench Verified's Python-only limitation.

Source

Source page Scale AI

↳ Follow the thread

SWE-bench Pro Ships as Multilingual Replacement for Abandoned SWE-bench Verified: 1,865 Tasks Across Python, Go, TypeScript, and JavaScript from 41 Repos Including Private Codebases | MindPattern