Hacker News new | ask | show | jobs
by anon223345 1141 days ago
Robots.txt is mostly ignored btw