AI Giants Back One Training Opt-Out for Publishers

OpenAI, Google, Microsoft and Anthropic have pledged to honour a single training opt-out for publishers, so a website can refuse the use of its pages for AI training with a shared instruction instead of a separate rule for every company. The pledge was announced on Friday, October 30, according to AFP, and it answers a complaint publishers have pressed for years: keeping content out of training sets currently means tracking down each company's crawler and writing a blocking rule for it by name.

Under the shared approach, a publisher adds a policy entry to the robots.txt file stating a preference about training use, and any participating company reads that entry the same way. The entry is built around a shared token, __usage_ai-training, set out in a draft submitted to the Internet Engineering Task Force by Kevin Kelley of Anthropic. The draft is an individual submission, not an endorsed standard, and it treats the token as a statement of preference about how content is used. It is not an instruction about crawling, so a page can be kept out of training while still being fetched for other purposes.

What the shared signal changes, and what it does not

Today, a site that wants out of training has to know every crawler by name. OpenAI uses GPTBot, Anthropic uses ClaudeBot, and Google uses Google-Extended for training, with other companies running their own crawlers besides. A publisher that misses a name, or that does not know a crawler exists, leaves a gap in its protection. The shared entry closes that gap for the companies that adopt it, because a publisher writes the preference a single time and every participating company interprets it the same way. Newcomers that adopt the convention inherit the same reading without the publisher having to update its file.

The limits are just as specific. The setting applies only to future training of models, and it does not prevent bots from crawling a site otherwise, Microsoft said in a blog post. Content scraping is allowed by default until a publisher changes the setting, and the signal does nothing about material already collected into existing datasets or trained into existing models. Robots.txt has always worked on trust, so a scraper that ignores the file today will ignore this entry too, and the announcement changes no legal position on training with content that was fetched anyway.

Different timelines for the same pledge

The four companies have not promised the same start date. Anthropic said it has "already started" rolling out support, while OpenAI said it will roll the signal out "in the coming months" without naming a date. Microsoft will begin honouring the preferences from February 1, 2027, as its Bingbot crawler revisits pages, giving publishers the firmest date of the group. Google aims to support the shared control sometime next year, with its crawler documentation narrowing that to early next year. A site that opts out through the shared entry without mentioning Google-Extended will be treated by Google as blocking Google-Extended, while a site that allows Google-Extended can still appear in Gemini answers.

Microsoft went further than the shared entry by adding a separate page-level control for grounding, the step that pulls live web pages into chat answers rather than training model weights. Meta, meanwhile, welcomed the idea of a common signal but has not said it will implement this token, according to reports of the announcement. Opting out also carries a trade-off publishers will have to weigh: Microsoft and Google said in separate blog posts that refusing training use could reduce the chances that a publisher's content surfaces in AI apps and experiences.

Why publishers pushed for this

The announcement lands against a long fight over training data. The New York Times sued OpenAI and Microsoft in December 2023, alleging millions of its articles were used without authorization or payment to train ChatGPT, and the two companies have invoked fair use, an exception to copyright law, in response. Publishers have also watched bot traffic surge as AI companies race to gather training material, and many smaller outlets lack the staff to maintain a separate blocking rule for every crawler that appears.

Other responses have already emerged elsewhere. Internet infrastructure provider Cloudflare has blocked AI bots by default on new customer sites since July 2025, and a standard called Really Simple Licensing already allows publishers to charge bots that visit their pages. The shared entry now gives publishers a simpler tool, but its reach depends on adoption beyond the group that announced it. The draft behind the token expires next May unless it is revised or adopted through the standards process, and the working group behind it is expected to keep refining the text. Until then, the pledge is a voluntary convention among rivals, and its test will be whether sites use it and whether other AI companies follow.