Hacker News new | ask | show | jobs
by mikemotherwell 2144 days ago
Other way around. It was invented to stop crawling. Indexing is still technically allowed even when blocked by robots.txt from crawling.