The biggest names in AI are backing a new way to opt out of scraping
Major AI companies including OpenAI, Google, Microsoft, and Anthropic are adopting a unified standard to allow website owners to opt out of AI scraping. This new 'policy token' will be implemented via robots.txt files, simplifying the process for publishers to manage their content across multiple platforms.
Why it matters
This move addresses growing tensions between content creators and AI firms regarding intellectual property and data usage rights.
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually.
The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website’s robots.txt file, which has long signaled which web crawlers can and can’t scrape a site — whether for search indexing, archiving, or AI training. Currently, sites have to signal their AI training preferences separately to each AI company.
Also covering this story
One other newsroom covered this event. We read that version too.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in