[alex_young]

> What gives you the right to load their content on terms other than theirs?

robots.txt for one thing.

[TeMPOraL]

robots.txt is a gentlemen's agreement between a site and search engine running automated crawlers. It has no meaning for regular visitors.