|
|
|
|
|
by NorwegianDude
929 days ago
|
|
I think search engines should be opt-in, not opt-out. Just linking to something is fine, but search engines also takes content and uses it to earn money. One example is the cards on Google where they extract content and shows it alongside ads. Nothing else I can think of works like this. Generally if you take the work of others without being allowed and use it to make money you'll end up in trouble. Now, for most people the trade-off is worth it, but it should not be the default. It should be a quick opt-in using robots.txt. |
|
If you put your site on the public internet, you are effectively opting in. In addition, robots.txt is trivially easy to configure if you really don't want your site to be crawled by specific parties' crawlers.
The news corporations absolutely want their sites crawled, to the point that if Google unilaterally stopped crawling their sites they would run to the courts to file lawsuits.