Y
Hacker News
new
|
ask
|
show
|
jobs
by
account42
15 days ago
Anubis is easy, just use a whitelisted user agent or a headless browser if some sites disable that - you need one to index web app abominations anyway. Cloudflare and Google reCaptcha are bigger problems.