[batmaniam]

Because of Dark patterns and Javascript. JS makes it hard to scrape unless you have enough processing power to run a browser headless for millions of product websites, and without bombing their sites too hard.

Then there's dark patterns where sites will literally tell you one price on the front page, but then at checkout tack on a ton of fees, and this is only AFTER you sign in, which means your price scraping bot has to have an account for millions of services that implement this pattern.

I don't believe this is something tech can solve, it's a people problem. The only way I can see this being a thing is if the government formalizes it in law or something, where each retailer has to submit the price to some centralized API where consumers can then pull from.

[burntoutfire]

> Because of Dark patterns and Javascript. JS makes it hard to scrape unless you have enough processing power to run a browser headless for millions of product websites, and without bombing their sites too hard.

I believe that, at least in Poland, it works the other way around? The price comparison site (ceneo.pl) is so dominant that the vendors have to actually feed it its listings in order to stay in business, as very many people just buys goods through ceneo, and pick whichever store has the good at lowest price at the moment.