Hacker News new | ask | show | jobs
by eesmith 1398 days ago
You assumed people don't give a crap about robots.txt, otherwise you wouldn't have been surprised.