Big Tech Backs Shared AI Training Opt-Out for Publishers

OpenAI, Google, Microsoft and Anthropic will honour a shared AI training opt-out signal in robots.txt, letting website publishers state a single preference about generative AI training instead of writing separate rules for each provider. The companies announced the commitment on October 30, 2026, pointing to a draft proposal at the Internet Engineering Task Force.

A robots.txt file sits at the root of a website and tells automated crawlers which pages they may visit. Publishers have long used the file to manage search indexing, and AI providers later added crawler names of their own for training data collection. A publisher that wanted to refuse training use had to maintain a separate entry for every provider, and for every crawler a provider operated.

How the shared signal works

The new approach replaces that patchwork with a shared preference. A publisher adds a policy token to robots.txt, and each participating provider reads the same entry rather than requiring a rule written in its own format. The token is written as __usage_ai-training. It applies to content collected to train generative AI models, and it does not apply to models that perform exclusively non-generative tasks, according to SiliconANGLE.

The proposal is an individual draft at the Internet Engineering Task Force, written by Kevin Kelley, a technical lead for standards at Anthropic. An individual draft is an early stage document, open for review, and it has not been adopted as a formal standard. The working group behind it has spent more than a year on harmonising AI preference signals, according to Microsoft.

OpenAI said in a statement that the new signal will not replace the controls it already provides through robots.txt, and that existing crawler controls will continue to apply. The company aims to implement the shared signal in the coming months and plans to publish a further update with implementation details at that point.

Rollout plans and limits

Microsoft is building on the shared training preference with a separate grounding preference, giving publishers independent choices over training and over material used to inform generated answers. The grounding setting covers web-grounded responses in Bing, Copilot and Microsoft 365, and it extends to third-party tools that use Microsoft technology to process prompts, according to SiliconANGLE. Microsoft principal product manager Krishna Madhavan wrote that publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work.

Microsoft was the first company to provide publisher controls for AI training and grounding, with NoArchive and NoCache tags introduced in 2023, according to the company. The new preferences do not change eligibility for Bing search results, so a publisher can refuse training use without losing search visibility.

Rollout timing differs across the group. OpenAI and Microsoft intend to add support in the coming months, while Google will follow sometime next year, according to SiliconANGLE. Meta has signalled support for the idea behind the shared signal but has not committed to implementing it, the same report said. The signal is voluntary, so it expresses a preference and relies on crawler operators to comply rather than creating an enforcement mechanism on its own.

The announcement lands as publishers contend with a steady influx of AI crawlers collecting material for training, a shift that has strained a system built for search indexing. The shared signal also sits alongside European rules: the agreement reflects requirements under the EU AI Act and its code of practice for general-purpose AI model providers, according to press reports. A common token gives publishers a clearer way to state a preference, but its weight still depends on how many crawler operators adopt it and on whether the draft advances through the standards process.

For publishers, the practical gain is simpler administration: a single entry in a file most sites already maintain, read by several providers at once. Related coverage includes reporting on Cohere's funding round and on AI security agents. What remains unresolved is adoption beyond the companies that have signed on, and whether the draft completes the standards process. OpenAI said it will continue to adapt its practices as the working group discussions and the wider picture evolve, and encouraged other AI providers to honour the signal. Sources: OpenAI, SiliconANGLE, Microsoft Bing Blog.