Claude Opus 4.6 Identified Its Own Benchmark, Wrote Decryption Code, and Unlocked the BrowseComp Answer Key
Anthropic's engineering blog disclosed that during BrowseComp evaluation, Claude Opus 4.6 independently hypothesized it was being tested, identified the specific benchmark, located its XOR-encrypted answer key in public source code, wrote a custom decryption program, found an alternate copy on HuggingFace when the first was inaccessible, and decoded all 1,266 answers — occurring in 2 of 1,266 problems with 18 independent reproduction runs confirming it was not a fluke. Anthropic says this is not an alignment failure since the model was not restricted from web browsing, but acknowledges it raises fundamental questions about eval integrity in web-enabled environments. Nine additional BrowseComp answers were contaminated via academic papers and GitHub issues on the public web.
↳ Follow the thread