Google joins OpenAI and Anthropic in promising new scraping controls - Engadget

Why it matters

Major AI developers — Google, Anthropic, OpenAI, and Microsoft — have agreed to honor a new robots.txt directive that lets publishers block the scraping of their content for generative AI training. The proposal, authored by Anthropic's Kevin Kelley and under consideration at IETF, replaces the cumbersome current system in which site owners must write separate rules for each individual crawler with a single common signal. Publishers add a dedicated user-agent section for AI training, and can carve out exceptions, such as allowing a press-kit page to remain scrapable. The directive does not block crawling entirely or remove content from search results, and it cannot undo scraping that has already occurred — it only governs future collection. Anthropic already supports it, Microsoft and Google plan to in 2027, and OpenAI within coming months. The story matters because it gives publishers a standardized, practical way to opt out of AI training data collection, and signals rare coordination among competitors on content-usage norms.