[JohnFen]

The primary purpose of robots.txt isn't to deny access. That's just a sideline. The intended purpose is to do exactly what this aura proposal does: to provide guidance to crawlers as to what parts of the site are valuable to crawl. That's why it's voluntary: it's main reason for existing is to benefit the crawlers in the first place.

In that light, I guess your proposal makes a certain amount of sense. I don't think it addresses what a lot of web sites want, but that's not necessarily a bad thing. Own your niche.

[OsmanDKitay]

You re right that not every website needs this today. My bet is that this becomes essential for any site that wants to be a verb (a place to do actions), not just a noun (a place to read content), in the emerging agent driven web. Thanks for the thoughtful discussion.

[paulryanrogers]

Are you thinking of sitemap?

[JohnFen]

No, sitemap serves a similar but different purpose. The design goal of robots.txt is to guide crawlers to the parts of the site that are worth crawling. That it's used as a very weak method of access control is a hack.