Hacker News new | ask | show | jobs
by jrwr 1633 days ago
They do respect robots.txt matter of fact, it will remove the website outright and most of its archives will be hidden until that website's robots.txt is offline.

Their documentation about it is rather crap right now, but its in several of their FAQs about it

1 comments

Respecting robots.txt would mean not saving pages blocked by robots.txt. Not just hiding.