[Too]

If I understand parent correctly, the restriction flag is opt-in? This turns copyright around completely, expecting every small content producer to implement something they likely are not aware of and secondly do not technically understand.

[Kim_Bruning]

At very least robots.txt is from 1994; it has been part of the web almost from the start (web became public in 1991, so within 3 years).

Claiming ignorance here would be just a little bit disingenuous.