[nerdralph]

I highly doubt there is no other technically feasible option to block the AI bots. You end up blocking not just bots, but many humans too. When I clicked on the link and the bot block came up, I just clicked back. I think HN posts should have warnings when the site blocks you from seeing it until you somehow, maybe, prove you are human.

[goobatrooba]

I'm sure there are many solutions for many problems, but expecting a small Foss development team to know or implement them all is rather unreasonable.

I think the world gains more if the VLAN team focuses on their amazing, free contribution to the world, than if they spend the same time trying to figure out how to save you two clicks.

We all hate that this is happening, but you don't need to attack everyone that is unfortunately caught up in it.

[overfeed]

> I highly doubt there is no other technically feasible option to block the AI bots.

If you have discovered such an option, you could get very wealthy: minimizing friction for humans in e-commerce is valuable. If you're a drive-by critic not vested in the project, then yours is an instance of talk being cheap.

[thresh]

I'm all ears on how we can fix it otherwise.

Keep in mind that those kinds of services: - should not be MITMed by CDNs - are generally ran by volunteers with zero budget, money and time-wise

[nerdralph]

First off, don't block the first connection of the day from a given IP. Rate limit/block from there, for example how sshguard does it.

I've seen several posts on HN and elsewhere showing many bots can be fingerprinted and blocked based on HTTP headers and TLS.

For the bots that perfectly match the fingerprint of an interactive browser and don't trigger rate limits, use hidden links to tarpits and zip bombs. Many of these have been discussed on HN. Here's the first one that came to memory: https://news.ycombinator.com/item?id=42725147

[thayne]

> don't block the first connection of the day from a given IP.

The bots come from a huge number of IP addresses, that won't really help that much. And it doesn't solve the UX problem either, because most pages require multiple requests for additional assets, and requiring human verification then is a lot more complicated than for the initial request.

> For the bots that perfectly match the fingerprint of an interactive browser

That requires properly fingerprinting the browser, which will almost certainly have false positives from users who use unusual browsers or use anti-fingerprinting.

> use hidden links to tarpits and zip bombs.

That can waste the bot users resources, but doesn't necessarily protect your site from bots. And Also requires quite a bit of work that small projects don't have time for.

Unless there is a prebuilt solution that is at least as easy to deploy and at least as effective as something like anubis, it isn't really practical for most sites.