[ronsor]

I know companies that already solve it.

[wredcoll]

I mean... knowing how to solve it isn't the trick, it's doing it a million times a minute for your firehose scraper.

[udev4096]

Anubis adds a cookie name `within.website-x-cmd-anubis-auth` which can be used by scrapers for not solving it more than once. Just have a fleet of servers whose sole purpose is to extract the cookie after solving the challenges and make sure all of them stay valid. It's not a big deal

[fc417fc802]

Requests are associated with the cookie meaning you can trace and block or rate limit as necessary. The cost of solving the PoW is the cost of establishing a new session. If you get blocked you have to solve again.

[creata]

Why is spending all that CPU time to scrape the handful of sites that use Anubis worth it to them?

[vhcr]

Because it's not a lot of CPU, you only have to solve it once per website, and the default policy difficulty of 16 for bots is worthless because you can just change your user agent so you get a difficulty of 4.