Hacker News new | ask | show | jobs
by xnx 137 days ago
That's what robots.txt is for.