[AlexandrB]

They try to block your bot because Google is a monopoly and there's little to no cost for blocking everything except Google.

This isn't a "natural" monopoly, it's more like Internet Explorer 6.0 and everyone designing their sites to use ActiveX and IE-specific quirks.

[luckylion]

One possible answer: pay them for their trouble until you provide value to them, e.g. by paying some fraction of a cent for each (document) request.

[BobaFloutist]

Cool, you wanna solve micropayments now or wait until we've got cold fusion rolling first...?

[luckylion]

You wouldn't have to make them micropayments, you can pay out once some threshold is reached.

Of course, it would incentivize the sites to make you want to crawl them more, but that might be a good thing. There would be pressure on you to focus on quality over quantity, which would probably be a good thing for your product.

[BobaFloutist]

>You wouldn't have to make them micropayments, you can pay out once some threshold is reached.

Believe it or not, this is a potential solution for micropayments that has been explored.

[immibis]

I could even pay a fixed amount to my ISP every month for a fixed amount of data transfer.