[dewey]

Some companies want you to list their products on your page (usually with some kind of affiliate deal attached) but don't have a tech team to implement a feed or an API. In that case you end up in a situation where you have to scrape the data yourself with permission.

[danpalmer]

Pretty much exactly this. We're no affiliates as we do our own fulfilment, but essentially we partner with companies to offer their products on our site, and we take a margin from them.

We sometimes do this with feeds, but feeds aren't great for stock change latency which is important for us (thin stock levels on wide range of products, more out of stock issues). Scraping ensures we at least have the same stock latency that direct customers see, and we manage the risk on how frequently to scrape.

Most of these companies don't have in-house engineering capabilities. For Shopify based merchants we don't have to scrape, we use the Shopify API, but otherwise scraping is the only real solution.