[mullingitover]

> I think search engines should be opt-in, not opt-out

If you put your site on the public internet, you are effectively opting in. In addition, robots.txt is trivially easy to configure if you really don't want your site to be crawled by specific parties' crawlers.

The news corporations absolutely want their sites crawled, to the point that if Google unilaterally stopped crawling their sites they would run to the courts to file lawsuits.