Hacker News new | ask | show | jobs
by canyp 11 days ago
I second this. My website exposes a cgit and 99% of the traffic now is AI scraping the sources, but the load is nowhere near DoS territory. And this is running on the cheapest VPS I could find.

Not saying I'm not annoyed by the scraping; I am looking to block them, but I'm also not going to put the site behind the gatekeeper. If anything, Cloudflare must love AI scraping now for the same reason AV companies love malware.

Now, if you are running a PHP stack...yeah, maybe that's the problem right there.
3 comments
lxgr 11 days ago
Is there actually any plausible theory why "AI" would repeatedly scrape the same sites? Are there that many competing, completely independent AI labs? Is it cheaper to repeatedly scrape than to buffer the scraped data locally? (I find it very hard to imagine that it's easier to deal with changing/disappearing content than it is to stand up such a cache.)
link
jack_pp 11 days ago
If you ask an agent to check sources / function definitions of open source packages it will wget / curl it
link
GoblinSlayer 10 days ago
It's an AI generated scraper that scrapes nonstop.
link
ern_ave 10 days ago
> 99% of the traffic now is AI scraping the sources

I wonder if we should stop fighting this and instead create an API specifically for this purpose? Or, a central repository that you could send your data to and say to anyone wanting to scrape, "safe yourself some time and just get my data from this other place"

link
canyp 10 days ago
The thing though is that they are extremely idiotic. They are constantly, recurringly, scanning the same files, I suppose out of FOMO that a line might have changed. I don't know what a special API solves, especially because HTTP already has etags to save you from re-downloading the whole damn file over again. But these bots don't care. The extent to which they don't care is such that, after I temporarily took cgit down for kicks, they'd get 404s and still repeatedly ask for the sames files days on end.
link
account42 10 days ago
The PHP stack isn't even the problem, it's having unauthenticated requests getting past the cache in the first place, something that most sites should be able to prevent.
link