[fc417fc802]

Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go this route.

[prologic]

PoW schemes like Anubis don't work. Increasingly bots are using headless browsers and are basically able to solve captchas, proof-of-work(s) and basically bypass all any any attempts to block them. It's becoming impossible to stop bots from hammering your sites/services for unwanted traffic.

[inigyou]

There's one or a small number of actors who are DDoSing the internet right now, and those are the ones you care about. What's the purpose for blocking non-DDoS bots?

[stephen_cagle]

Seems like there is no POW "solving" if the POW in question is literally solving some sort of mathematical puzzle. It has a bounds and we can just scale the bounds depending on how much abuse we are seeing.

I've never understood why that isn't more common honestly. It would be cool if I could actually transfer POW from one domain to another as well. Probably wouldn't stop scrapers, but it would likely stop low margin fraud type things.

[m00dy]

The most recent captchas that involve basic reasoning can block majority of bots today.

[prologic]

Name one, that an LLM can't solve. The only real mechanism that continues to still work (but doesn't scale) is "Human in the loop".

[fc417fc802]

You are asserting that they don't work without explanation or evidence. Meanwhile it isn't clear why they wouldn't and indeed they appear to accomplish the stated goal of severely rate limiting scrapers.

[codethief]

IIRC there was a blog post here on HN recently explaining in detail why Anubis doesn't keep out AI bots.

[fc417fc802]

Yes the default config can be trivially bypassed. When it becomes a problem the UA exceptions can be removed. It's no different than how difficulty can be scaled up and down as circumstance demands.

[gib444]

> You are asserting that they don't work without explanation or evidence.

Oh, no, that isn't how it works — you made a claim first (albeit indirectly), with no evidence. It's on you

[inigyou]

Evidence: they solved many people's problems

[fc417fc802]

That's not how it works. There are cases where established norms or common sense dictate a certain default assumption. This is one of them.

Still, I'll humor your absurd request by reminding you of the many success stories that have repeatedly made the front page of HN.

Beyond that we have cryptocurrencies. If you have knowledge of a generalized solution for defeating PoW schemes then why are you posting here instead of making yourself a billionaire?

[prologic]

Surely I don't need to explain how headless browsers work, bot proxies and distributed crawlers to you do I? If you haven't experienced this first-hand, I can understand. Maybe it warrants a blog post. But I can assure you, these don't work at scale, they might only stop the "less sophisticated" bots and maybe (just maybe) some unwanted spam.

[fc417fc802]

What do the browsers being headless or the traffic proxied have to do with anything? Each client still has to solve a PoW challenge regardless. That mitigates the (possibly unintentional) DoS by making it exorbitantly expensive.

[gruez]

>I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites.

???

Unless you're browsing around with the googlebot user agent string, you should be getting turnstile challanges at most, not blocks. And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.

[amatecha]

Try using Firefox with "resist fingerprinting" enabled, on OpenBSD, if you want an idea of how insidious Cloudflare's increasing prevalence is for the openness of the web. Inexplicable 403 responses for entire domains for no apparent reason, no "captcha", just blocked outright. Heck, it happens on Linux too, and even without "resist fingerprinting" enabled.

[ipaddr]

Happens with older chrome browsers as well. Last supported windows 7 version. Started this month.

[gucci-on-fleek]

> And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge.

You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge.

> If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's.

Cloudflare blocks legitimate users itself sometimes [1].

[0]: Unless you run into a bug, but Anubis is open source, so you can always submit a patch upstream. I've done this myself, and I can confirm that it's relatively straightforward.

[1]: https://news.ycombinator.com/item?id=43329320

[gruez]

>You are guaranteed to pass an Anubis challenge eventually [0]

That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap.

>Cloudflare blocks legitimate users itself sometimes [1].

I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.

[gucci-on-fleek]

> That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap.

Yeah, I really have no idea why Anubis works right now: residential proxies are far more expensive than compute, yet the bots seem to have no problem obtaining millions of residential IPs, but they give up on even short-ish Anubis challenges.

> I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.

Yeah, I don't like the Cloudflare challenges, but in the past 5 years I've only had it outright block me once, and that fixed itself after 15 minutes. And I use Firefox on Linux with various privacy extensions, so my browser probably appears at least moderately suspicious.

Whereas I've been trapped in impossible ReCaptcha loops quite a few times, which is still better than vague error messages that magically go away when I switch to something not running Linux. So I'll begrudgingly accept that Turnstile is the least user-hostile product on the market right now.

[inigyou]

Because the (singular) global ddos adversary is using a very dumb scraper, even dumber than wget --mirror

[raincole]

The only reason Anubis works right now is that it's not very commonly used and the bots are not optimized to bypass it (yet).

[inigyou]

The bots that are problems right now are not the ones that run Anubis challenges. There is no universal anti-bot, only ones that work at certain times.

[fc417fc802]

> unlike humans, their time is dirt cheap.

On the contrary I only want to visit a few pages on a given site and have an entire laptop at my disposal. Meanwhile for bots efficiency is key. A serious scraper (ie the type of actor that actually causes material problems for site operators) is performing tens or hundreds of pages loads per second per core spread across thousands of sites. Making a single page load take even half a second of cpu time is a massive win for the site operator.

[benhurmarcel]

I’ve been blocked by Turnstile without possibility of challenge, no way to access nor know why. And it was with standard iOS and Windows machines and no VPN or anything. It happens.

[inigyou]

Yesterday Cloudflare told me I was a bot and couldn't access this.

[m00dy]

yeah, googlebot user agent thing was a thing maybe around 10 years ago but I don't think it's working today. Maybe if you start the http session from gcloud, it might still work.

[timpera]

Please don't use Anubis, it makes visiting websites very difficult (often multiple minutes wait times) on old hardware and low-end smartphones.

[altairprime]

As things stand right now, the current web may not be possible to maintain for old and low-end phones given the costs imposed by AI training crawlers.

Please propose an alternative to both Cloudflare and Anubis, that shields websites against inhuman traffic without frequent operator intervention (or otherwise negates the capacity costs they pay for AI crawling) and is compatible with low-end smartphones.

Certainly, I imagine Anubis would be interested in adopting it if it’s effective!

[timpera]

The solution is building simpler and more efficient websites. You won't win the arms race, so make it irrelevant. Any random VPS hosting a static website can already serve millions of requests an hour.

[expedition32]

Great you now made the internet unusable for half the planet.

[shimman]

Sounds like something the government should regulate and own then rather than relying on corporate benefactors that all profit off of malware advertising.

[altairprime]

That will be about as effective as attempts to block soccer streams in Spain are, and I don't think it'll pan out as a viable solution without worldwide enforcement backed by BGP bans of providers that continue to host non-compliant sites after notified. It's technically possible to consider a future where RIPE might withdraw your BGP assignment if you host advertising malware on your website, but ultimately that's just as dystopic as the DMCA is today, only scaled up to worldwide impacts and drama.

[altairprime]

AI is absolutely going to make the internet unusable for half the planet if we don't find a better way; it's happening in realtime and I hate to see it. I would love to know of a better way, but everyone's so busy clamoring against effective solutions without proposing anything else that would work.

Doing nothing doesn't work. Doing Cloudflare isn't acceptable. Doing Anubis makes the Internet unusable "for half the planet". I'm on your side morally, but site operators can't afford to be idealists in the face of AI crawling bills.

What do you, does anyone, suggest that hasn't been tried, or adopted, or considered? What's left that will defend operators against AI's traffic flood that isn't eating a thousand doller crawler bill, paying Cloudflare, or locking out half the planet?

[fc417fc802]

Personally I think that's a better outcome than the alternative.

However another option is to support both. Have a challenge page that requires the visitor to select one of several options. Cloudflare could be one of those.

[gobip]

What's the alternative? We're stuck, either cloudflare or anubis

[r_lee]

what kind of hardware would take minutes to complete the checks?

[timpera]

It really depends on the difficulty level chosen by the website. I'm testing on both a Xiaomi Redmi 5 and my university library's computer.

[matheusmoreira]

> Please consider installing one of the many PoW schemes such as anubis

Why not go all the way and mine monero instead of just completely wasting the work?

[akersten]

because then someone will complain that you're stealing their CPU cycles or something, on the website they chose to visit and execute

[matheusmoreira]

"Stealing" CPU cycles is exactly what PoW bot protection is doing. The whole point is to add cost to the bots so they decide it's too expensive and give up.

The only difference is the cycles are getting converted into heat now. They could be getting converted into monero instead. It's still heat but at least creator got some money for it.

[m00dy]

Not sure why Anubis is getting so much hype on HN, but honestly, it is not the solution. A real solution would use behavioral modeling. Most browser fingerprinting issues are already largely solved anyway.

[bornfreddy]

As someone who is regularly blocked by CF and G - no, they are not. Even worse, they should not be, because that would mean the complete end of privacy online (not that we are far from that). Anubis works because at scale it wastes bots' resources (time mainly).

[m00dy]

I think you’re confusing bots that crawl a website with bots used to launch DDoS attacks.

[inigyou]

it works because the global ddos adversary doesn't run javascript

[stephen_cagle]

I'm genuinely baffled that this is your lived experience. I wonder if you only look at a very restricted set of websites?

[fc417fc802]

PoW is a reasonable solution as a fallback when other metrics flag a client.

[microtonal]

How? First, they can solve the Anubis challenge with native code, so they can solve them faster than genuine users. Second, the cost is nothing compared to training LLMs, plus they will just move the work to the residential proxies that they have access to (so, someone is paying through their TV's electricity bill).

Anubis only work(s|ed) great for a while when crawlers were not prepared for these challenges. Security through obscurity.

[fc417fc802]

> they will just move the work to the residential proxies that they have access to

The proxies I am familiar with do not offer arbitrary code execution. I think you're thinking of a botnet.

Regarding native code, the current crop of solutions seem to work well enough for now. Ultimately a challenge response protocol should be standardized and browsers should ship a native implementation. In the meantime WASM likely gets you close enough to native.

[microtonal]

Regarding native code, the current crop of solutions seem to work well enough for now.

I just quoted a toot in another submission, adding it here since it is relevant:

We apologize for a period of extreme slowness today. The army of AI crawlers just leveled up and hit us very badly. [...] It seems like the AI crawlers learned how to solve the Anubis challenges. [...] However, we can confirm that at least Huawei networks now send the challenge responses and they actually do seem to take a few seconds to actually compute the answers. It looks plausible, so we assume that AI crawlers leveled up their computing power to emulate more of real browser behaviour to bypass the diversity of challenges that platform enabled to avoid the bot army.

https://social.anoxinon.de/@Codeberg/115033790447125787

[neya]

Adding the link to GitHub here if anyone is curious:

https://github.com/techaroHQ/anubis