Ars Technica Analysis: Google AI Overviews Wrong 10% of the Time — Millions of Incorrect Answers Per Hour

Ars Technica·medium signal

A New York Times/Oumi analysis using OpenAI's SimpleQA benchmark found Google's AI Overviews produce incorrect answers approximately 10% of the time. At Google's scale of 5+ trillion annual searches, this translates to tens of millions of wrong answers per hour. Google disputes the methodology, arguing SimpleQA itself contains incorrect information and prefers their 'SimpleQA Verified' variant with more thoroughly vetted questions.