Hacker News new | ask | show | jobs
by cbjoe 4971 days ago
Google indexes URLs despite measures such as robots.txt when these URLs are discovered by Google software including Chrome and their Toolbar.
1 comments
Evbn 4971 days ago
Robots.txt is about fetching content, it has noting do to with indexing URLs or anything which is part of the content at non-robots.txt restricted locations.
link