[Dylan16807]

This phrasing is very misleading. To bullet point directly from "possibility" to "standard" implies the standardization was a turning point where it could start being used. But it was massively used long before that. The standard is a side note that's barely relevant.

[reaperducer]

It only became massively used in 2019, when Google recommended it.

[Dylan16807]

Where did you get that date?

https://serverfault.com/questions/171985/how-can-i-encourage...

Here's a 2010 discussion about Google's explicit support, and I'm sure I could find earlier.

The thing google did in 2019 was submit it as a standard, nothing to do with adoption or starting to recommend. In that very post they said "For 25 years, the Robots Exclusion Protocol (REP) has been one of the most basic and critical components of the web" "The proposed REP draft reflects over 20 years of real world experience of relying on robots.txt rules, used both by Googlebot and other major crawlers, as well as about half a billion websites that rely on REP."

[reaperducer]

Where did you get that date?

On July 1, 2019, Google announced the proposal of the Robots Exclusion Protocol as an official standard under Internet Engineering Task Force.[8]

https://en.wikipedia.org/wiki/Robots.txt

[Dylan16807]

That is not when they started recommending it. It would be nice if you acknowledged the rest of my comment, I even quoted from the [8] reference.