Hacker News new | ask | show | jobs
by seren1 1610 days ago
TIL that indexing and crawling are different, and robots.txt prevents Google from crawling but not indexing.