Reddit
Web Search for AI Agents Hitting a Wall — Google Index Restrictions + Cloudflare Bot Blocking
A 317-upvote r/LocalLLaMA thread with 191 comments documents the convergence of two forces throttling AI agent web access: Google restricting its free search index and Cloudflare's Bot Fight Mode challenging AI user agents at every gateway (default-on since the April 2026 Webflow migration). The UK CMA has proposed mandating Google give publishers control over AI Overview usage. Practitioners report web-search-dependent agent workflows are degrading rapidly, with no clear replacement for the open web crawling model.
Source
↳ Follow the thread