Defenses against abusive AI scrapers

Even more collateral damage

The AI hype has been ongoing for a while now, and the ecological impact is bad. I mean, old and hazardous coal and nuclear power plants are being re-started, and new, untested nuclear power plants are planned to be constructed - just for the energy hunger of powering these large language and other models. However, there is more collateral damage than that.

At the time of this writing (January 2025), there are many reports of web sites being brought nearly down - or at least being loaded so badly that the slowdown becomes noticable to normal visitors - because of abusive webscrapers presumably collecting training data for AI/ML models. Some/many of these webscrapers, either due to errors in programming, ignorance, or intention to ignore the wishes of web site operators, ignore the long-standing standard of reading and parsing robots.txt (standardized in RFC 9309).

What is robots.txt?

This is the first line of defense. Adding a robots.txt to a website provides instructions for well-behaved crawler bots on what not to include.

Because the de-facto standard (before becoming an RFC) is really old (dating back to 1994), there is a lot of support for creating, parsing, and respecting these webscraping conditions in standard software: e.g., Hugo, the static website generator used for this page, has template support to generate robots.txt. Currently, I don’t use the template, but a static robots.txt in the source of this web page to experiment more easily with filters on user-agent strings, and am combining strings from multiple sources and what I see in my own server log.

Why is it not enough?

Unfortunately, there is quite a bit of bad or even abusive behavior connected to massive crawling efforts right now, starting with bots that simply ignore the robots.txt as standard declaration. Because of these, we have to add other layers of defense.

Legal defenses

Various jurisdications, include the European Union, now have provisions for declaring opt-in to or opt-out from automated data mining. On this site, I use a declaration in plain language that should hopefully make my intention clear. However, it is still unclear how enforceable this is. Legal proceedings will take time to work this out, and then will probably have to relate to standards on how to to such declarations in a scalable manner - possibly making robots.txt more legally binding in the process…

Technical defenses

So what do we do with bots/webscrapers that neither honor the machine-readable robots.txt standard nor any plain-language legal declaration? There are multiple options.

Bandwidth limits

The first technical defense is limiting bandwith. That obviously depends on the stack the webserver is running on, but the simplest path is most probably via the web server that is already in use. For this small personal web page, I use nginx (inside a simple Alpine docker container that includes the statically compiled website, all-in-one) behind Traefik (which does the TLS termination with Let’s Encrypt certificates). Both can apply rate limiting for incoming requests, but nginx provides more flexibility, including limiting the output bandwith instead of limiting only incoming requests per time period.

Because my virtual machine running this site is (intentionally) small, I am now limiting output bandwidth per client with a simple nginx config file snippet added to the self-contained docker container. This wasn’t necessary before the influx of all those AI/ML data scrapers that now seems to be more than 90% of all traffic to the site.

UPDATE 2025-04-21: I am now using much more aggressive parallel connection, rate of requests, and transfer volume rate limits, because in the last week before this change, obnoxious AI scraper bots - completely ignoring robots.txt and falling into the trap - became so aggressive within the link maze that they started overloading my Traefik ingress CPU consumption and therefore effectively performed a DDoS on my other services on this host. I am now also using built-in Traefik rate limiting with fairly simple limits in the docker-compose config of this container:

This combination of Traefik and nginx rate limiting brought CPU consumption from 100% back down to below 60% for all my services combined (including Nextcloud and Vaultwarden, which were previously unusable for my authenticated users) and load average from over 10 to around 3 at the time of this writing - most of the load spent on TLS termination by Traefik This is how stupid and bad those scraper bots have become. I call them out as illegal denial of service at this point.

UPDATE 2025-06-02: Because the above still caused unneccessary CPU load, I added the (in hindsight) obvious mitigation/optimization to serve the linkmaze and gzip bomb via HTTP instead of HTTPS. There’s really no reason for wasting CPU cycles on my end to encrypt and integrity-protect bogus data. Because this was slightly more annoying to configure in Traefik than I initially expected, let me share my current docker-compose lines for separating HTTP and HTTPS traffic on this service:

The real downside of this additional optimization is that now I need to specify the HTTP-to-HTTPS redirect rule for every other standard service on the same Traefik instance, because the previously global default redirect I had in traefik.yml cannot, as it seems, be overridden for individual services. So if you also want to do this to save CPU cycles caused by abusive web scrapers, you will also need to change every other service config as well…

IP-level blocking

A second technical defense could be blocking on an IP level. fail2ban, crowdsec, and other similar log analysis software can automate this with the standard web server logs based on e.g. rate of requests from IP addresses or ranges. Integrating should be fairly easy, as most servers are likely already using such tools to block brute force login attempts to authenticated services (I certainly am).

Unfortunately, (some of) the most abusive crawlers no longer use their own infrastructures, but botnets for their scraping. Therefore, web services are hit by a flood of different client IPs with different (probably faked/randomized) user agent strings. Blocking on an IP level is therefore going to be even more painful. Maybe that means going all-in and sharing abusive scraping signals through centralized “bad IP” collectors like crowdsec and getting clients that act as bots in such abusive botnets banned from many services at once.

UPDATE 2025-06-04: In addition to the distributed botnets, it turns out that some bots actually still use their own infrastructure addresses, although distributed within that larger network. asncounter is one good tool to discover the source of IP packets classified by ASNs. Running it in tcpdump mode on the respective Docker interface, I got the following output over a (hopefully representative) time period of a few hours:

JKU Linz is in the list primarily because I run an external automated monitor (Zabbix, to be specific) there for some of my services to get notified when anything becomes unreachable.

As also documented here, Huawei bots seem to actively fake their user agent strings in addition to not honoring robots.txt. I got too annoyed by this kind of abusive behaviour and am now hard blocking Huawei (and Alibaba, for good measure, due to their similar misbehaviour documented elsewhere) with nftables reject rules:

Tarpits

Tarpitting refers to a mix of deliberately slowing down the output of some service to slow the crawler/bot and to keep them locked into an endless series of auto-generated (and slowly delivered) links to follow. The intention is to consume more resources on the abusive bot side than the delivering server, and to prevent or at least delay further abuse of other services.

This idea is not new, and has been applied at a time when webcrawlers were used to scrape email addresses for spam lists in what feels like decades ago. Now we turn to this technique again, and software like Nepenthes or Quixotic implement different aspects of a tarpit for webcrawlers.

On this site, an automatically created random link maze is served under a sub-URL that is disallowed for all bots in robots.txt. Those that do not respect it will be led into the link maze.

Both the link maze and the poisoned data are subject to strict bandwidth limiting to not consume significant resources (on either side, actually). The intention is only to slow them down.

Poisioned data

This last technical defense (for now) is actually more specific to the AI/ML hype: generating fake data that presumably causes negative effects for the abusive webcrawler real goals (e.g., training ML models) and might therefore put them off scraping data that they are actually forbidden from getting in the first place. The hope here is that, like with tarpits, these webcrawlers will be made to respect the clear statements in robots.txt.

On this site, randomly created data is served under a sub-URL that is disallowed for all bots in robots.txt. Those that do not respect it will ingest random, fake data.

gzip bombs to inconvenience stupid webcrawlers

UPDATE 2025-05-01: The distributed abusive crawlers keep being obnoxious, and even when slowing down the number of parallel connections and individual connections per IP through rate limiting, they bog down my Traefik TLS termination by overwhelming it through sheer numbers of clients (presumably steered by a central botnet controller). I therefore added a feature to Quixotic to randomly inject a static URL into the linkmaze pages. On this nginx server config, that means the linkmaze will include links to a 10MB file called resume.html that is actually transparently delivered from a gzip of about 10GB of dev/zero. Clients that support content-encoding: gzip transfers will receive the 10MB gzipped content and decompress (client-side) to 10GB, hopefully exhausting the available memory of stupid crawlers - and by definition, any bot that ignores robots.txt is considered stupid.

I recommend NOT clicking this link from your normal web browser, which will support gzip content-encoding with high probability. For testing, I tend to use

which will stream directly to disk and not cache in RAM. For a different solution, see e.g. this.

Summary

It is quite unfortunate that we seem to see a kind of arms race here: AI/ML companies trying to grab all training data they can, and doing so repeatedly and without regard to clearly communicated rules, legal frameworks, and technical collateral damage on the one hand, and web site operators and their intellectual and operational costs on the other. The active defenses would not be necessary if established standards like robots.txt were being honored. These defenses require even more manual effort and computational resources to set up and keep running. We can only hope that webscrapers will be sufficiently inconvenienced with the technical defenses so that they will start to behave rather than ratchet up their abuse by trying to work around those specific defenses, furthering the arms race.