[toofy]

> The bots have a different model -- they assume that the website is adversarially "hiding" its content.

this should give us pause. if a bot considers this adversarial and is refusing to respect the site owners wishes, thats a big part of the problem.

a bot should not consider that “adversarial”

[chii]

> refusing to respect the site owners wishes

should a site owner be able to discriminate between a bot visitor and a human visitor? Most do, and hence the bots treats it as a hostile environment.

Of course, bots that behave badly have created this problem themselves. That's why if you create a bot to scrape, make it not take up more resources than a typical browser based visitor.

[danaris]

> That's why if you create a bot to scrape, make it not take up more resources than a typical browser based visitor.

Well, right; that's the problem.

They take up orders of magnitude more resources. They absolutely hammer the server. They don't care if your website even survives, so long as they get every single drop of data they can for training.

Source: my own personal experience with them taking down my tiny browser game (~125 unique weekly users—not something of broad general interest!) repeatedly until I locked its Wiki behind a login wall.

[expedition32]

This is like email were eventually 90% of it was spam and we all got spam filters.

[danaris]

Except that something effectively equivalent to spam filters will be utterly ineffective here.

Spam filters

- mitigate the symptom (our inboxes being impossible to trawl through for real emails)

- reduce the incentive (because any spam mail that isn't seen by a human being reduces the chances they'll profit from their spamming)

- but does not affect the resource consumption directly (because the email has already been sent through the internet)

Now, this last point barely matters with spam, because sending email requires nearly no resources.

With LLM-training scraper bots, on the other hand, the symptom is the resource consumption. By the time you see their traffic to try to filter it, it's already killing your server. The best you can hope to do is recognize their traffic after a few seconds of firehose and block the IP address.

Then they switch to another one. You block that. They switch to another one.

Residential IPs. Purchased botnet IPs. Constantly rotating IPs.

Unlike spam, there's no reliable way to block an LLM bot that you haven't seen yet, because the only thing that tells you it's a bot is their existing pattern of behavior. And the only unique identifier you can get for them is their IP address.

So how, exactly, are we supposed to filter them effectively, while also allowing legitimate users to access our sites? Especially small-time sites that don't make any money, and thus can't afford to buy CloudFlare or similar protection?

[expedition32]

Bandwidth isn't free. And god knows the bots ain't paying.