Hacker News new | ask | show | jobs
by kevinpet 4971 days ago
robots.txt is not about indexing. It's about crawling.