OpenAI, Google, Microsoft, and Anthropic Back an AI Training Opt-Out

My Medium friends can read this over on Medium.

Recently, I wrote that AI companies should pay rent for my content. Every month, for every site they crawl and keep using.

On October 30, OpenAI, Google, Microsoft, and Anthropic announced something else. A line for robots.txt that tells them not to train their AI on a site.

No money involved.

The signal

robots.txt is the small file on every website that tells bots what they may crawl. The new entry looks like this:

User-Agent: __usage_ai-training
Disallow: /

That means: don't use anything on this site for AI training. You can also allow single folders, a press kit, for example.

One detail is easy to miss. The usual User-Agent: * line doesn't count. If a robots.txt only has that, it says nothing about training. The new token has to be named.

And it's a preference. Bots can still visit. They're just asked not to train on what they find.

Also, it isn't a finished standard yet. It's a draft at the IETF, the organization behind many internet standards. Someone from Anthropic submitted it on the same day. The working group hasn't adopted it.

Who's in

Four companies say they'll follow it. Voluntarily. Nobody enforces it.

Microsoft has a date: February 1, 2027. Google says early 2027. OpenAI says "in the coming months." Anthropic says it has started.

Meta says it supports publishers' ability to express preferences. Whether it will honor this signal, it didn't say.

Apple, Perplexity, xAI, Mistral… no commitment yet. At least, I haven't found one.

Training only

The signal covers training. Not answers.

In my rent piece, the bigger problem was the answers. People get them from the AI and never click through to the site the answer came from.

Google's AI Overviews and AI Mode aren't touched by the new robots.txt signal. They have their own setting in Search Console. Microsoft splits it as well: one setting for training, one for "grounding," meaning when Copilot and Bing use a page to answer a question.

Both companies say opting out doesn't change a site's normal search ranking.

At Google, there is a catch, as far as I can tell. Opting out with the new signal counts as blocking Google-Extended. And Google says sites that allow Google-Extended are eligible to appear, with clicks, in Gemini's answers. So saying no to training probably means saying no to those clicks too.

The old rules for each company's bot stay as well. GPTBot, ClaudeBot, Google-Extended. The list in robots.txt doesn't get shorter.

Money

None of the four mentions payment. Microsoft says it plainly: the settings "do not establish licensing or payment arrangements."

The draft even says private agreements can override it.

And RSL, the standard that lets a site put a price into robots.txt? I mentioned it last time. None of the four mentioned it now. The RSL Collective lists no AI company that licenses through it, as far as I can see.

Same day, OpenAI published another post, titled "Fair use benefits everyone." In it, OpenAI calls the New York Times lawsuit a pursuit of an "undeserved payday."

Make of that what you will.

Downsides

• It's a draft. The details can still change before it becomes a standard. If it becomes one.
• It's voluntary. A bot that ignores it faces nothing.
• It's four companies. Plenty of other crawlers aren't on the list.
• It doesn't touch the AI answers, which is where the clicks go.
• And at Google, it may cost a site its spot in Gemini's answers.

The Bottom Line

So now there's a way to say no to four AI companies. Starting 2027, mostly.

A way to say "yes, for this much" still doesn't exist.

If you're not already, follow me on Medium.