Making it easier for websites to express AI training preferences

Why it matters

OpenAI has announced it will honor a shared signal within robots.txt that allows website operators to state their preferences about the use of their content for foundational generative AI training and communicate those choices to multiple AI providers at once. Under the current setup, operators had to identify individual crawlers and maintain separate rules for each provider; a common signal reduces that burden. OpenAI said it aims to implement the signal in the coming months and will share implementation details later, and it stressed that the new mechanism will not replace its existing robots.txt crawler controls, which will continue to apply. The company is joining other participants in the Internet Engineering Task Force's AI Preferences working group in announcing the shared signal, framing the move as strengthening its participation in the IETF standards discussions. OpenAI described the change as a practical step toward an open web where publishers have simpler ways to express their choices, and it encouraged other AI providers to honor the signal as well.