[BinaryIdiot]

> I think evidence goes the other way. You can ask web crawlers not to index certain pages with robots.txt even if it would be better for their business if they did.

I disagree. robots.txt is almost always used for hiding pages that shouldn't be exposed to the internet and are useless to expose. For example you don't need a robots.txt crawling the html document you're statically serving to prove domain ownership for Google Apps.

Everyone wants the most views on their content as possible so the incentive is to let as many things as possible be indexed therefore using robots.txt is limited as much as possible. It would not be good for search engines to crawl the things put into robots.txt.