[sjwright]

And if you have some way to identify yourself as a potential competitor to google and not some jackass trying to scrape email addresses or spam comments forms, I'm all ears.

[greglindahl]

A majority of the websites that blekko, a google competitor, contacted to ask for robots.txt access ignored us.

[sjwright]

I agree, it's a difficult conundrum. It sucks.

[greglindahl]

There are worse barriers to entry for a search engine! DMCA take-downs... Right to be forgotten... click history...