[1gn15]

As with what others had said, this is less of a robots.txt and more of a sitemap.

The issue with this is that website owners don't want to do this. Take Reddit removing the API for example. Everyone just switched to scraping the Reddit website instead for any remaining third party clients.

Yes, APIs were supposed to be a compromise to lower the resources needed on both sides, but Reddit's stock price is linked to the value of "their" data, so...

Alternatively, malicious website owners may make incorrect Aura files to mislead user agents. Then we're back to screen scraping as the ground truth, because behaving like a human is the best way to avoid discrimination.

[hypertele-Xii]

If the website owner is malicious, the site's content is untrustable either way.

[OsmanDKitay]

The Reddit example gets right to the heart of it: when a platform loses control over its data and how it makes money, it just shuts the door on open access.

And that s really what aura is trying to address. If we dont figure something out the web could easily end up just being a handful of big ai sites, and all the smaller, independent sites might just fade away.

The goal with aura is to give control back to the people running the websites. It s not about blocking ai but about giving sites a clear, standard way to say "here s how you can work with me meaningfully...". This means an agent can do something specific and useful without costly, aimless scraping, and it lets site owners build cool, new features just for ais.

and you re right to worry about malicious manifests but that s a trust problem. A site that lies in its aura file would get a bad reputation fast just like a phishing site does now.

At the end of the day aura is a bet that we can build an open, capability based web where site creators can join in on the AI revolution on their own terms. Time will tell if it's the right technical answer, but it s a conversation i think we absolutely need to have to keep the web diverse and creative.