[lloyddobbler]

I've worked with two shopping search engines, and interestingly, scraping sites was one of the things they did to build up their inventory as well. The big difference being, they simply organized the products into a searchable format, then sent traffic to the ecommerce site and let them handle the checkout . What you're doing is arguably more complex.

(They also prioritized the feeds that were sent to them directly by retailers above the scraped items feeds - thus prioritizing paid listings, similar to the Google SERPs - so a different business model entirely.)

That being said, a very cool concept - and agreed that, given the relatively-small number of ecommerce platforms out there, scraping then erving them up seems pretty scalable. Interested to see how it goes.

[razvanr]

At least one of them might have been using our technology in the backend, especially if they're one of the top 5 shopping search engines.

The downside to feeds is that they become obsolete very quickly, especially if the product is popular. Products sell out very quickly, retailers lose money on traffic they can't onboard and shoppers get frustrated.

Thanks for your thoughts!

[lazyjones]

> At least one of them might have been using our technology in the backend, especially if they're one of the top 5 shopping search engines.

Which of these "top 5 shopping search engines" have you worked with? You don't seem to mention any on your website.

> The downside to feeds is that they become obsolete very quickly, especially if the product is popular. Products sell out very quickly, retailers lose money on traffic they can't onboard and shoppers get frustrated.

Feeds are the only way to keep up with frequently changing listings from large retailers (apart from doing live API requests) since scraping is several orders of magnitude slower. Amazon gives selected partners incremental feeds, scraping their millions of products takes days.