[AussieWog93]

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots?

WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.

Are you using lambdas or something like that to serve a blog?

Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?

[Ndymium]

In my case it's my Forgejo instance. My blog is static files with a limited amount of pages to crawl and indeed it's no issue at all. Forgejo is a dynamic service with an infinite amount of pages to discover and it runs git in the background to generate (some of) the pages. Tons of bots can easily overwhelm my small server. I could hide the repositories, but they're open on purpose (it's open source after all).

I have it guarded by a simple cookie check now, which seems to work well enough. Only a small subset of bots pass it. It likely kills search engine discoverability, but that's the tradeoff I have to make.

[inigyou]

That seems fine. You are using a proportionate response to an actual problem.

[extra88]

I have a personal site on a shared host that was recently suspended for excessive CPU use caused by incessant AI bot crawling. I don't have a problem with crawling per se, there's just so many and they're so bad at it.

[Bender]

For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript. Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any option people consider should be tested on a test server. People could just point and laugh if they so desired and that works for me too.

[Bender]

As of now roughly 4661 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers.

I could improve this by moving most of the blocking to a demo site so the article is easier to access. I will do that. I will also update some of the nginx ACL's to what I am currently using that explain to the botters what is out of place.

[Barbing]

Bender, even got yourself a great username. One of the most responsive authors in memory.

[Bender]

Thankyou. It's a fun topic for me. Hopefully others will find it interesting to tinker and experiment with as well.

[Bender]

As of now 6622 real humans made it through and a small number of what look to be bots. Curious if any of these people will try their own experiments on this topic.

[milkshakes]

i guess the thing i'm most confused about is what is the higher level goal here. 1 in 8 humans on the planet are experiencing the "web" through chatgpt alone. many have migrated to purely agentic workflows.

is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is the strategy?

[Bender]

I like those ideas. If people are truly operating purely through LLM's then I am fine with being partitioned from them. The LLM operators will provide their reality and their truth. This is all for low trust internet-wide access.

For smaller higher trust communities I don't do any of this. Rather we use basic authentication to keep bots and strangers off the services all together and then regular user accounts on forums and such. That is where strategy comes into play and has been working well for some time.

[milkshakes]

i guess what i'm curious about is where you would draw the line, and why.

how do you define a user agent? is it desirable or not for users to be able to discover and access resources and communities on the internet using the tools, formats, and workflows that they prefer?

are search bots desirable?

the internet wayback machine?

how do you feel about browser extensions and greasemonkey scripts?

[Bender]

I do not define user-agents, they announce themselves and present a particular behavior. I let them define their behavior and I respond accordingly.

I have no need for search bots personally. If I had a complex site I would build my own search feature. If it was a commercial site I would pay for advertising on popular sites that were of related categories.

Wayback is fun to play with but it's just a toy to me. There is no concept of domain ownership. Most of the snapshots of the domain I am using were from when someone else rented it. They claim to respect robots.txt but that is not entirely true. They crawl and save content even if a site says not to and as soon as the site is offline and robots.txt is no longer accessible they will display all the archived content. This means a person has to park their domain on a server containing a robots.txt that matches their intentions.

Browser extensions are mostly invisible. What is obvious to me is when people are using "reader" applications as web clients. They should be concerned more than me. They are reading random sites run by strangers using apps that may or may not have been battle hardened, reviewed by third party penetration testers and so on. Some of them are vibe coded in unsafe languages.

[razorbeamz]

Do you have a source for those numbers?

[jolux]

ChatGPT has over a billion monthly active users.

[Bender]

I am an active user of Claude but I do not live my life through that thing. I try to get it to answer questions that it does not want to answer. The only questions it seems to answer without hesitation are technical in nature. In the entire existence of my account I have started 9 chats.

More interesting to me was that Alexa on Amazon Prime answered all my questions without hesitation or disclaimers but once I have my answers I close that tab.

[somenameforme]

This is definitely a 2026 type comment, but I'm very surprised that it's so "low." GPT is the most well known brand and it's the one that seemingly 100% of kids use to "assist" on their homework and other exercises from grade school to college. One would think that alone would already take you well over a billion.

[razorbeamz]

This does not mean that all billion of them are using ChatGPT as their main interface with the internet.

[watwut]

I am monthly active user, absolutely do not experience web through ChatGPT alone and frankly, dont mind ChatGPT being cut off stuff. AI companies forcing themselves as a middle man is not a good thing.

Whoever using purely agentic worflow is not my concern. Just like people who read only physical book, learn only from podcasts or watch only netflix movies.

There is zero reason for the rest of us to worry about filling free data to companies that try to make themselves monopolies.

[dminik]

I wonder what the click through rate is for the source links. Based on what I've been hearing, it's very low. And that's if the particular LLM/interface/answer even bothers to add them.

If the bots are eating your site traffic and users, what's the point of allowing them in?

[AussieWog93]

Not pointing and laughing, just wondering why (since I see so many posts about this). Sounds like it's just for the thrill of the game, which is completely valid.

[Bender]

Thrill of the game is a great way to put it.

[inigyou]

I wish it was framed as "silly ways to block bots" then so that people wouldn't take it as a recommendation

[nubinetwork]

At home, I can't afford a 40gbit connection and a server to match... getting DDoSed by a handful of malicious google cloud VPSes running nmap and various web vulnerability scans can easily impact performance on low end hardware.

Yes, in the grand scheme of things, waiting a few additional seconds to check my local imap server (because my DMZ gets bogged down) isn't a huge deal, but that doesn't mean I have to like it, or allow it to continue.

[inigyou]

Are you getting 40gbits of bot traffic?

[mkj]

It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things.

Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine!

But getting 2.7 million requests/day from residential proxy IPs was a bit taxing for the old uwsgi/cgi programs (that's average 30/s), making other sites on the server suffer. It was also getting close to 1TB/month which is the VPS quota.

So I ended up turning them off, one more cut towards the centralised internet. Sure I could put a cache in front, but there are probably millions of combinations of dynamic urls in the VCS tree so caching mightn't help much anyway, not worth the effort. I'm sure the web server could be tuned better too, but again it's a waste of time.

[Barbing]

30/s is preposterous. Are these loser companies who don’t want to cache on their end or are they getting unique info per request or could it really be that many different origins or…?

[yjftsjthsd-h]

> Are these loser companies who don’t want to cache on their end

Google couldn't be bothered to cache; I would guess this before anything else.

https://drewdevault.com/blog/Google-has-been-DDoSing-sourceh...

[Barbing]

> each IP address still clones the same repositories 8-10 times per hour.

Dumb! When was this fixed and don’t say never :)

[inigyou]

That just sounds like a CI system. Surely almost every one of us is responsible for at least one CI process that doesn't cache dependencies.

[mkj]

There are a lot of urls for the bots to follow. Yesterday's logs had 2.2M unique URLs out of 2.7M requests (more unique than I was expecting!)

Typical traffic:

  https://hg.ucc.asn.au/dropbear-fuzzcorpus/annotate/default/fuzzer-preauth/23f6b74342011f35e657f06e9398ad339360d7e9 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/comparison/e67e5df225e5/fuzzer-verify/b5f57a2bb85ab7ef957e4979fd202b091de27bd8?revcount=8 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/log/72337e3dcccf/fuzzer-preauth_nomaths/6b1f390d26b08759983937727e10ab90aa7b03f2?revcount=15 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/log/4cdb539e4b20/fuzzer-preauth_nomaths/713e5cd4f73276f00a5d6685cc16d4565ad3815a?revcount=96 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/diff/tip/fuzzer-client_nomaths/540778ea81a915af6e898fd9658c5d9d6594c40a?revcount=128 
  https://hg.ucc.asn.au/dropbear-fuzzcorpus/diff/0ef1e51e583f/fuzzer-client/65ac44bdf4868866def70f40542dd1b5ca447310?revcount=30

(I've moved and kept the fuzz corpus now, but other viewvc repos were suffering similarly)

[psd1]

I have compiled a list of vibecoders who have impressed me with their consideration for other people. Here it is:

[inigyou]

None of the above. They're just really retarded bots. Nobody knows why they exist or what they gain from really stupid scraping.

[inigyou]

> 1TB/month

Switch VPS providers, you can easily find higher numbers or "unlimited".

[drusepth]

Bots routinely overwhelm my public-facing websites. The last few times I've turned on "attack mode" in Cloudflare (which I do when bot swarms get particularly nasty), traffic dropped by at least 20k requests/minute (which is huge, considering my human traffic is around 3/4 of that).

[Nemi]

I run an ecommerce auction website on wordpress that is highly dynamic and can't be cached. Bots are a waste of cpu. I even turned off "prefetch" requests as I found that (android in particular) browsers will prefetch every auction item on the page as the user scrolls down the summary page looking at items.

[pibaker]

What about "our hosting bill shot up 500% from bot traffic and went down to the baseline the moment we put our site behind cloudflare or anubis"

[c2h5oh]

My dedicated server comes with 2TB/month data included. Humans use up about 2% of that. If I didn't block bots I'd run out mid-month.

[inigyou]

On a dedicated server? It must be somewhere like Korea, where bandwidth is extremely expensive? Usually, dedicated servers are already expensive enough that they throw in unlimited bandwidth for free. And when not, its usually more like 200TB/m

[chlorion]

My cgit instance gets hammered so hard that my CPU use is raised to 100% and the entire server grinds to a halt. I have to set a cgroups CPUQuota limit with systemd to make the server usable again.

Also, they hit so hard that nginx returns 502s and times out for real user half of the time, and when the site does load it takes a few seconds to load.

This is all on a high performance single core vultr server, not the cheapest vps but not the fastest, but should be plenty fast enough.

[loloquwowndueo]

> WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.

Yes we know - I still had my site knocked out a couple of times by overeager bots. They can be really abusive.

[AussieWog93]

Wow, that's nuts. What sort of site was this? Mine don't get anything near this level of attention!

[loloquwowndueo]

Just a personal blog, and I have to admit it had a misconfiguration that was my fault - nonexistent urls would redirect to the index. Still, dumb scraper bots would 1) not realize there was a redirect loop and kept trying every possible url, and 2) they were doing so aggressively enough that it brought the site down a couple of times. A well-behaved crawler would have slowed down and noticed the loop, never had issues with crawlers for years until the advent of ai.

[jeroenhd]

One of my issues had to do with pure bandwidth, dozens of bots were requesting the same image over and over again adding random crap to the query. My server was being slow for me during crawling peaks.

> An actual proper static site would likely 10x that.

A lot of the stuff I host isn't static. Any non-static content also gets overloaded pretty quickly. Hosting something like a Git server can turn pretty resource intensive if shitty bots start diffing every commit with every other commit.

I'm generally lazy when it comes to blocking IPs, but when I do run into an issue, I just blanket block IP ranges. When that doesn't work, ASNs get blocked. So far I haven't had to take most measures mentioned here but I probably will once the broken bots start coming from botnets using residential IP addresses.

[imglorp]

Egress costs money.

[cuchoi]

Hundreds of fake applicants to a position