[immibis]

You don't get to tell site owners what to do. The actual facts on the ground are that they're trying to block your bot. It would be nice if they didn't block your bot, but the other, completely unnatural and advertising-driven, monopoly of hosting providers with insane per-request costs makes that impossible until they switch away.

[AlexandrB]

They try to block your bot because Google is a monopoly and there's little to no cost for blocking everything except Google.

This isn't a "natural" monopoly, it's more like Internet Explorer 6.0 and everyone designing their sites to use ActiveX and IE-specific quirks.

[luckylion]

One possible answer: pay them for their trouble until you provide value to them, e.g. by paying some fraction of a cent for each (document) request.

[BobaFloutist]

Cool, you wanna solve micropayments now or wait until we've got cold fusion rolling first...?

[luckylion]

You wouldn't have to make them micropayments, you can pay out once some threshold is reached.

Of course, it would incentivize the sites to make you want to crawl them more, but that might be a good thing. There would be pressure on you to focus on quality over quantity, which would probably be a good thing for your product.

[BobaFloutist]

>You wouldn't have to make them micropayments, you can pay out once some threshold is reached.

Believe it or not, this is a potential solution for micropayments that has been explored.

[immibis]

I could even pay a fixed amount to my ISP every month for a fixed amount of data transfer.

[threeseed]

> The actual facts on the ground are that they're trying to block your bot

Based on what evidence.

[immibis]

based on them matching the user-agent and sending you a block page? I don't know what else to tell you. It's in plain sight.