Why The Concept of Data Poisoning Annoys Me

Below is a description of the Nightshade data poisoning utility from their own website (source: https://nightshade.cs.uchicago.edu/whatis.html).

It is extremely important to be able to protect our work from being stolen, and against low effort near reproductions being made and claimed to belong to someone else. The law is not protecting us as it should be, and we are left to fend for ourselves. I agree this is a problem, and this is the problem that data poisoning aims to address. How then could the concept of data poisoners annoy me? Certainly I should be a fan, and even more, perhaps I should use my graphics programming / computer science / research knowledge to help advance the technology?

Absolutely not, and I will tell you why: It makes people believe their work is safe, when it isn’t. That makes it dangerous, and it is not fixable, unfortunately.

For data poisoning to work, there needs to be some mechanism of perception where a machine would fail, but a human would succeed.

Data poisoners work by finding instances of this being true in specific models. That doesn’t protect from other models though, or from future models – and the future goes on for a long while (for better or worse).

Captchas are similarly doomed to ultimately fail, but the difference there is we can keep updating them in an arms race to keep up (essentially captchas being the adversary part of a GAN). With data poisoning, as soon as you need an iteration on the poisoning tool, this is a sign that the previous poisoning is no longer effective against the current models, which means all “protected” data is now unprotected.

If we want lasting protection, we need to find some perceptual mechanism that a human has but a machine does not have, and could never have. We can think about where something like that could come from. Anything that is biology based, chemistry based, or physics based can essentially be done by a machine.

Beyond that only lies concepts like the soul and free will. These things are untestable, and thus have no actual effect that us engineers can exploit to separate humans from nonhumans, or biology from machines. This is why I say the problem is ultimately unfixable.

Yeah sure, we can try to make things have prohibitive cost or difficulty to deal with – this is the age old battle in security after all – but even if we get that, the real problem is that poisoned data is not universally poisoned and does not stay poisoned. That is a shame, but that’s reality.

Did you enjoy this article? You might also enjoy this one:
Why The AI Determinism Argument Annoys Me

Also, here’s a paper on arxiv about the weaknesses of a specific type of data poisoning, even for models it was meant to poison.
https://arxiv.org/abs/2406.12027