Y
Hacker News
new
|
ask
|
show
|
jobs
by
JohnFen
317 days ago
No, sitemap serves a similar but different purpose. The design goal of robots.txt is to guide crawlers to the parts of the site that are worth crawling. That it's used as a very weak method of access control is a hack.