Hacker News new | ask | show | jobs
by wareya 4092 days ago
The whole point of robots.txt is that there are pages which people may hit that bots can't. What are you on?