[mkj]

It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things.

Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine!

But getting 2.7 million requests/day from residential proxy IPs was a bit taxing for the old uwsgi/cgi programs (that's average 30/s), making other sites on the server suffer. It was also getting close to 1TB/month which is the VPS quota.

So I ended up turning them off, one more cut towards the centralised internet. Sure I could put a cache in front, but there are probably millions of combinations of dynamic urls in the VCS tree so caching mightn't help much anyway, not worth the effort. I'm sure the web server could be tuned better too, but again it's a waste of time.

[Barbing]

30/s is preposterous. Are these loser companies who don’t want to cache on their end or are they getting unique info per request or could it really be that many different origins or…?

[yjftsjthsd-h]

> Are these loser companies who don’t want to cache on their end

Google couldn't be bothered to cache; I would guess this before anything else.

https://drewdevault.com/blog/Google-has-been-DDoSing-sourceh...

[Barbing]

> each IP address still clones the same repositories 8-10 times per hour.

Dumb! When was this fixed and don’t say never :)

[inigyou]

That just sounds like a CI system. Surely almost every one of us is responsible for at least one CI process that doesn't cache dependencies.

[mkj]

There are a lot of urls for the bots to follow. Yesterday's logs had 2.2M unique URLs out of 2.7M requests (more unique than I was expecting!)

Typical traffic:

  https://hg.ucc.asn.au/dropbear-fuzzcorpus/annotate/default/fuzzer-preauth/23f6b74342011f35e657f06e9398ad339360d7e9 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/comparison/e67e5df225e5/fuzzer-verify/b5f57a2bb85ab7ef957e4979fd202b091de27bd8?revcount=8 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/log/72337e3dcccf/fuzzer-preauth_nomaths/6b1f390d26b08759983937727e10ab90aa7b03f2?revcount=15 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/log/4cdb539e4b20/fuzzer-preauth_nomaths/713e5cd4f73276f00a5d6685cc16d4565ad3815a?revcount=96 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/diff/tip/fuzzer-client_nomaths/540778ea81a915af6e898fd9658c5d9d6594c40a?revcount=128 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/diff/0ef1e51e583f/fuzzer-client/65ac44bdf4868866def70f40542dd1b5ca447310?revcount=30

(I've moved and kept the fuzz corpus now, but other viewvc repos were suffering similarly)

[psd1]

I have compiled a list of vibecoders who have impressed me with their consideration for other people. Here it is:

[inigyou]

None of the above. They're just really retarded bots. Nobody knows why they exist or what they gain from really stupid scraping.

[inigyou]

> 1TB/month

Switch VPS providers, you can easily find higher numbers or "unlimited".