News
Cloudflare's Default Block on Mixed-Use AI Crawlers Went Live September 15
The policy Cloudflare announced in July took effect today: crawlers that blend search indexing, agent retrieval and model training are now blocked by default on ad-bearing pages, and every configuration that blocks AI training also blocks mixed-purpose bots. It applies automatically to new customers, new sites from existing customers, and all existing free-tier sites; the legacy 'Block AI bots' toggle is deprecated the same day. Anyone running a retrieval agent against Cloudflare-fronted sites should expect fetch failures starting now unless the crawler declares a single purpose.
Source
↳ Follow the thread