|
|
|
|
|
by sounddust
5745 days ago
|
|
There's a huge difference between what Google is doing and what you're doing. 1) Google is caching pages for a specific purpose and ensuring that they aren't cached/scraped by others: http://webcache.googleusercontent.com/robots.txt By not excluding robots, you're opening yourself to all kinds of situations where you are responsible for draining revenue from the owner of the content, which leaves you liable to lawsuits. By contrast, the way that Google caches content and their rules surrounding it do not generally harm the copyright owner. 2) Google honors all robots.txt, no-archive meta-tags, and other indications that the author doesn't want the page to be cached. Is historious doing the same? |
|