Hacker News new | ask | show | jobs
by bdcravens 4978 days ago
You'd end up blocking all traffic then. When was was the last time you pulled robots.txt?