[danpalmer]

Wow, they've built a distributed Amazon listing scraping system – essentially a botnet.

As someone who has done a lot of web scraping and had to route around a lot of blocking (we have business contracts to allow scraping, but they don't stop over-eager sysadmins), this feels like a dream come true.

But I'd never actually want to use this for scraping and I'm not sure any informed user would agree to use this.

[voltagex_]

How do you get contracts to allow scraping? What kind of cost are we talking about?

[dewey]

Some companies want you to list their products on your page (usually with some kind of affiliate deal attached) but don't have a tech team to implement a feed or an API. In that case you end up in a situation where you have to scrape the data yourself with permission.

[danpalmer]

Pretty much exactly this. We're no affiliates as we do our own fulfilment, but essentially we partner with companies to offer their products on our site, and we take a margin from them.

We sometimes do this with feeds, but feeds aren't great for stock change latency which is important for us (thin stock levels on wide range of products, more out of stock issues). Scraping ensures we at least have the same stock latency that direct customers see, and we manage the risk on how frequently to scrape.

Most of these companies don't have in-house engineering capabilities. For Shopify based merchants we don't have to scrape, we use the Shopify API, but otherwise scraping is the only real solution.