Hacker News new | ask | show | jobs
by pseudalopex 1635 days ago
Respecting robots.txt would mean not saving pages blocked by robots.txt. Not just hiding.