[Ndymium]

Search engine crawlers generally respected robots.txt and limited themselves to a trickle of requests, likely based on the relative popularity of the website. These bots do neither, they will crawl anything they can access and send enough requests per second to drown your server, especially if you're a self hoster running your own little site on a dinky server.

Search engines never took my site down, these bots did.

[atonse]

Thanks for specifying the actual issue. As someone who hosts a bunch of sites, we're also seeing a spike in traffic, but we don't track user agents.

[OutOfHere]

Maybe stop using an inefficient PHP/Javascript/Typescript server, and start using a more efficient Go/Rust/Nim/Zig server.

[Ndymium]

Personally I'm specifically talking about Forgejo which is Go, but calls git for some operations. And the effect that was worse than pegging all the CPUs to 100% was filling of the disk with generated zip archives of all of the commits of all public repositories.

Sure, we can say that Forgejo should have had better defaults for this (the default was to clear archives after 24 hours). And that your site should be fast, run on an efficient server, and not have any even slightly expensive public endpoints. But in the end that is all victim blaming.

One of the nice parts of the web for me is that as long as I have a public IP address, I can use any dinky cheapo server I have and run my own infra on it. I don't need to rely on big players to do this for me. Sure, sometimes there's griefers/trolls out there, but generally they don't bother you. No one was ever interested in my little server, and search engines played fair (and to my knowledge still do) while still allowing my site to be discoverable.

Dealing with these bots is the first time my server has been consistently attacked. I can deal with them for now, but it is an additional thing to deal with and suddenly this idea of easy self hosting on low powered hardware is no longer so feasible. That makes me sad. I know what I should do about it, but I wish I didn't have to.

[OutOfHere]

It is why I require authorization for expensive endpoints. Everything else can often be just an inexpensive cache hit.