Waiter, there’s a watermark in my slop

David Gerard covers the new text watermarking in Anthropic’s LLM output: Anthropic watermarks AI output — and the AI bros yell.

Nobody who claims they can tell good writing from bad should care this much. If they’re not using the slopbot instead of doing the writing.

The watermarked text doesn’t even make it into the final writing project, if the LLM was just used for assistance, with methods such as Seth Godin’s AI tear down. So it shouldn’t really matter. Maybe Matt Birchler is right, and the reader’s desire to know when text is AI-generated is more important than the AI user’s desire for the opposite. Maybe Dave Winer is right in A rebuttal to Doc Searls’ piece about watermarks and the risk that a watermark might be detected could help discourage people from getting a long bit of writing from the bot, and then pasting it into a comment thread in a GitHub repo I run.

The bigger question, though, isn’t about the negligible difference between watermarked slop and somehow purer non-watermarked slop.

If LLM output can have watermarks, what else can it have?

Other identifiers. If Anthropic can get a 1-bit yes/no watermark into 200 words of text, how many words does an LLM provider need to encode a 32-bit identifier? That’s probably infeasible in the lengths of text that most people use LLMs for in practice. Maybe they would be able to find the authors of all those slop clone books on Amazon dot com, but for most business writing and article-length text it wouldn’t work. SynthID, described in Scalable watermarking for identifying large language model outputs, encodes a single bit of information by over-weighting token choice from one red list and one green list. Splitting the list into 2n categories would give you more bits but require a lot more text to encode it.

But what about just a few more bits? Now that governments and Big Tech know that encoding information into LLM-generated text is possible, what’s next?

• USA Citizen/non-citizen? This would be a obvious one for a company trying to get in good with the Federal government here.
• Age verification? I don’t know, they’re sticking age verification on everything now, someone will try it.
• Marketing-related data? The highest-earning 10% of people in the USA buy 50% of the stuff, so all kind of uses for a flag to show which side of the K-shaped economy the user is on.

Infringing or plagiarizing text. The big AI companies have a pretty well-defined political program, and can identify likely opponents to some or all of that program. Some of those opponents are consistent “AI vegans” in their own personal IT choices, but if someone is politically inconvenient and an AI user, well, it’s possible to tune the likelihood that an LLM’s output contains material straight out of the training set.

Could the LLM’s output contain deliberate infringement bombs or plagiarism bombs? Deliberately giving users some text that would get them in trouble later is not the kind of thing that a human developer would risk—too much risk that the commit messages would come out in discovery—but it is the kind of trick that a heavily “agentic” automated software process would come up with. An AI agent can already hack a gym to get its owner a spot in pilates class, so a program of compromising political opponents who are also users seems feasible.

Marketing side effects. Some LLMs are serving ads now, which means a lot of hard-to-predict ML-driven ad placements. And this stuff will be a lot weirder and more indirect than current projects like the attribution cartel, which is pretty clearly going to cause ML to come up with privacy-violating ways to juice the apparent results from Big Tech advertising. ML systems going for other goals are going to get a lot weirder. Meta’s ad ML is already doing pretty weird (and creepy) stuff to maximize engagement, and it’s only going to get weirder.

What happens when more ad-revenue-maxing ML is in the loop in more places? What if you can sell a renter’s insurance policy to user A by giving their friend, user B, some bad household tips that result in an expensive bill from their landlord? Consumer-facing LLMs are run by the same companies that find themselves in a desperate squeeze to keep raising ad revenue at startup-like growth rates. Corners will be cut. Other ways will be looked. Slop advice will reflect the need to achieve ambitious business goals, even at the user’s expense.

Anyway, the difference between watermarked slop and non-watermarked slop is tiny compared to some of the text that’s going to be in the LLM output. Enjoy.

Bonus links

Hundreds of Fake VPNs Are Flooding the Chrome Web Store by Ritoban Mukherjee. (Ad blockers are another category with similar problems. If you’re looking for browser extensions on the browser’s extension store, you’re doing it wrong. Start with a trusted IT news site.)

Representing Python paths using pathlib by Jake Edge. (One of the ingredients in upgrading a Python program from junk drawer script to maintainable. Two more: dataclasses instead of free-form dictionaries, and logging instead of just print() for debugging output)

AI is making everything worse by Garrett Graff. (One more: scammers have adopted “AI” enthusiastically and at scale. We already had a scam culture crisis before the “AI” boom, and the existing services help the scammers a lot more than they help the scam fighters. Related: Another ad safety* report)

Meta ran ads for an app promising to nudify female politicians by Vittoria Elliott and Matt Burgess. (File under “your company’s choice to advertise on Meta sites and apps is a political statement.”)

Young People Hate AI CEOs So Passionately That It’s Almost Hard to Believe by Joe Wilkins. Those are some appalling approval ratings, reflecting the massive swing in popularity the tech industry has experienced over the last decade, driven by concerns around data privacy, the purposeful decay of once-useful platforms, and the erosion of democracy. (Don’t forget online harassment and scam culture. When a normal person’s day-to-day experience of an IT brand is a flood of dick picks, lootbox-infested games, and rip-off dietary supplements, that brand is not going to be super premium.)