[Dwedit]

How do you build a search index in the days of Anubis pages everywhere?

[account42]

Anubis is easy, just use a whitelisted user agent or a headless browser if some sites disable that - you need one to index web app abominations anyway. Cloudflare and Google reCaptcha are bigger problems.

[majorchord]

https://roundproxies.com/blog/how-to-bypass-anti-bots/