[PaulHoule]

Robots.txt was created long before Google and before people were thinking about SEO:

https://en.wikipedia.org/wiki/Robots.txt

The scenario I remember was that the underfunded math department had an underpowered server connected via a wide and short pipe to the overfunded CS department and webcrawler experiments would crash the math department's web site repeatedly.

[abirch]

With the advent of AI and the notion of actually going to a website as being quaint: each website should have a humans.txt such as https://www.netflix.com/humans.txt or https://www.google.com/humans.txt

[EPendragon]

I have not heard of humans.txt before. It is apparently used for acknowledgement and crediting the dev team who created the resource.

[PaulHoule]

What everybody is missing is that AI inference (not training) is a route out of the enshittification economy. One reason why Cloudflare is harassing you all the time to click on traffic lights and motorcycles is to slam the door from some of the exit routes.

[LorenPechtel]

Yup. Robots.txt was a don't-swamp-me thing.

[EPendragon]

It is so interesting to track this technology's origin back to the source. It makes sense that it would come from a background of limited resources where things would break if you overwhelm it. It didn't take much to do so.