Hacker News new | ask | show | jobs
by eli 5465 days ago
Google can even index files listed in robots.txt (it just doesn't crawl them)