[yellow_lead]

> The exact adversary Anubis targets defeats it trivially.

Wrong, anubis stops mass crawling of web pages, by requiring a proof of work, which makes accessing these websites more expensive.

Anubis was not built to stop individual users with llms.

[drum55]

Claude Opus can make an optimized version of the proof of work solver that’s more than 10000x faster than the javascript one, in about 5 minutes time. Who is this stopping exactly? It’s the wrong tool in the wrong place.

[Macha]

Practically, the people indiscriminately scraping don't bother to do the work to bypass it or implement the POW test, which results in reduced CPU load for all the properties that were having trouble with scrapers before. Until scrapers start implementing it en masse then, it still serves its purpose.

[greyface-]

> Until scrapers start implementing it en masse

If it becomes widespread (as it has been doing), they will. Anubis' strategy only works while it remains a niche approach only adopted by a small number of sites.

[Macha]

And then people will move to a new solution.

I think this is a pragmatist vs idealist debate. The pragmatic answer is that Anubis solves a problem now, and so it will be used until either it doesn't or a better solution presents itself. The idealist approach is that it's obvious that there's ways to get around Anubis, and so some people argue from there that it shouldn't be used. But the only other alternatives being offered are to either eat up the costs (in server resources or engineering time), or to go behind cloudflare with its own tradeoffs.

[greyface-]

Anubis is the penicillin of web hosting.

Every individual practitioner has a strong incentive to overuse it, because it's extremely effective for them individually. Every additional practitioner that uses it increases the selection pressure on their collective adversary to develop resistance. Eventually, it reaches a tipping point and becomes ineffective. But the ecosystem-level impact of its historical use remains, and makes things worse for everyone.

Recently, the baseline expectations of the Web have shifted, and I need to enable JavaScript to read a non-CloudFlare-proxied simple HTML site like LKML. I do not expect this shift will revert once Anubis outlives its effectiveness.

[Macha]

Viruses do not have motivations, so it’s hard to assign blame to them. The companies running badly behaving AI scrapers are run by people with more of a mind than viruses, so please direct your complaints about the second order effects of their actions that way, rather than on their direct victims.

[Krutonium]

I do want to note that my website has Anubis, and does not require Javascript. You can configure Anubis to test in other ways, JS is just the default.

[drum55]

The barrier is a two sentence prompt to Claude to add a patch to curl that does 500MH/s to the 50kh/s the javascript version does on the same hardware, it only works because it’s obscure and largely irrelevant.

[inigyou]

Then they'll change what Anubis does.

[PunchyHamster]

That was true but with AI that could be automated pretty easily. Sure, not worth for random blog but random blog won't get the traffic anyway

[Kuinox]

The difficulty is increased when the server get heavy traffic.

[drum55]

Then the only people who can use it are the ones with a GPU implementation.

[Kuinox]

No, the value of scrapping the page is long gone and the scrapper would have moved on. The users would still access the page by waiting a bit.