|
|
|
|
|
|
by ezequiel-garzon
4489 days ago
|
|
|
The de facto standard robots.txt is pretty likely to be respected by Google, so it's fairly easy to stop their scraping your site. Yes, it is opt-out, bit I'd expect it to be. It may be quite frustrating for an upstart to be denied access while Google is explicitly allowed, but that's another matter. |
|
|