Hacker News new | ask | show | jobs
by nfhshy68 1398 days ago
Nothing, I don't make assumptions. That's something you do.
1 comments
eesmith 1398 days ago
You assumed people don't give a crap about robots.txt, otherwise you wouldn't have been surprised.
link