[terminalcommand]

Are you only crawling or also scraping the website?

If the sites in question only add an exception for googlebot and not other crawlers (e.g. Yahoo, bing, etc.) I would say that it is against the site owner's consent.

However if the site owner adds this exception also for other crawlers, you could argue that the site owner's intent of only allowing certain crawlers has not been made explicit. In that case you'd have a chance against the claims from the site's owner.

On the other hand Google could possibly sue you for using the user-agent "Googlebot".

The important question here is: would they? If you stay under the radar no one -even the courts- would bother.

PS: I am only a law student, I am not familiar with any laws/regulations/precedents governing this specific issue. I think from the site owner's perspective it's a grey area. From google's perspective brands and ip are established concepts in law. This is a student's very personal opinion at first sight, take it with a grain of salt :).

[jrvidal]

> On the other hand Google could possibly sue you for using the user-agent "Googlebot".

Genuinely curious: on what basis? Can you trademark (or similar) a user-agent?

[terminalcommand]

Google is a registered trademark, Googlebot may not be registered. However as it clearly is affiliated with Google, even if Google had not registered it they might pursue claims.

I am not familiar with U.S. law. If the courts in U.S. adopt a strictly formalistic approach on using names in a business context, then the results may be different.

However IMHO, Googlebot is clearly associated with Google and anyone who uses Googlebot as their User Agent is tricking the sites owner's into believing that the request was made from Google.

As for legal grounds, Google could sue on unfair competition.