Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Cloudflare is implementing a new policy that will block 'mixed-use' web crawlers from ad-supported sites by default starting in September 2026. This move aims to force AI companies to pay for content access and distinguish between traditional search bots and AI training agents.
Why it matters
This policy could fundamentally shift the economic relationship between AI developers and content publishers by enforcing stricter control over data scraping.
Cloudflare has just issued the AI industry a new deadline to separate the web crawlers used for traditional search purposes, like Google Search, from those used for AI agents and training. Starting on September 15, 2026, Cloudflare’s default settings will block “mixed-use” crawlers from any pages that host ads, the company announced on Wednesday.
The article presents both Cloudflare's strategic move and Google's existing opt-out mechanisms without taking a side.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in