Google Joins OpenAI And Anthropic In Promising New Scraping Controls
Major AI companies including Google, OpenAI, and Anthropic have agreed to adopt a new robots.txt directive to manage how their models scrape web content. This standard allows publishers to opt-out of AI training data collection through a unified signal.
Why it matters
This represents a significant shift in how the internet manages intellectual property and data rights in the age of generative AI.
The new robots.txt directive will prevent the companies from scraping content for generative AI training.
tadamichi/Shutterstock Add Engadget on Google: Preferred Source Google Discover Google , Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive, which will allow publishers to specify their preferences when it comes to the scraping of their content for generative AI training. The companies have decided to adopt a proposal under consideration at IETF, the open standards organization for the internet, written by Anthropic technical lead for standards Kevin Kelley.
Also covering this story
One other newsroom covered this event. We read that version too.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in