Y
Hacker News
new
|
ask
|
show
|
jobs
by
paulryanrogers
309 days ago
Are you thinking of sitemap?
1 comments
JohnFen
309 days ago
No, sitemap serves a similar but different purpose. The design goal of robots.txt is to guide crawlers to the parts of the site that are worth crawling. That it's used as a very weak method of access control is a hack.
link