[nostrademons]

I think you got the wrong RFC number:

http://tools.ietf.org/rfc/rfc3938.txt

No mention of robots.txt, nor a section 6.

[jfr]

Sorry, the correct number was 3986.

RFC 3986 - Uniform Resource Identifier (URI): Generic Syntax