OpenAI's 'Deployment Simulation' Replays Real Past Conversations Through a Candidate Model to Predict Misbehavior Before Release

OpenAI·high signal

OpenAI introduced a pre-deployment evaluation method that takes recent de-identified deployment conversations, strips the old model's reply, and regenerates it with the candidate model to estimate real-world failure-mode frequency — analyzing ~1.3M conversations spanning GPT-5 Thinking through GPT-5.4. It claims models can't distinguish simulated from real traffic, giving a more realistic preview than adversarial red-teaming, and the technique now extends to agentic coding via simulated tool calls. For builders this is a reusable eval pattern: replay your own production traces against a new model/prompt before you ship it.