Hacker News new | ask | show | jobs
by elliewithcolor 1031 days ago
Yes it’s legal. Just don’t check the price ever 2 seconds from 800 locations.

A simple way would be a headless browser [1]

But there are also hosted tools that work like a website builder.

The best way is: keep it simple and keep back (check once an hour or day and not every minute).

Many shops use Schema.org markup. So if they support it, you don’t have to write it for every site.

You could also use a library that works with raw html and css. Then you could just use css selectors for extraction.

[1] https://www.atlantbh.com/building-a-dynamic-crawler-with-pup...