[fooock]

Yes, in the practice people sometimes don't want to be polite with webmasters, and choose not obey robots.txt rules. Thanks for the suggestion!

[itsmefaz]

Exactly, your service could definitely be used as an alternate to parsing robots.txt (which traditionally is in xml) to a more standard json parsing. Along with the advantages that comes with making it REST.