[sph]

> One way Google maintains their monopoly is that many websites block all bots except for the Google indexing bot.

No, I don't think that's true. I am writing a web crawler, not for search purposes, and I haven't seen preferential treatment for GoogleBot compared to others. Sure, some might be banned outright (though bad crawlers just ignore robots.txt and do whatever they want), but in most cases new bots have the same access rights than GoogleBot.

Also, your sentence doesn't pass the sniff test: you claim Google has better access than all other crawlers; but robots.txt is solely in the hands of the webmaster. How does Google coerce most website owners to block other bots? There is no conspiracy at play here.

[nubela]

Try LinkedIn, or Facebook, or Twitter, or Crunchbase.