Hacker News new | ask | show | jobs
by ausjke 2068 days ago
this is all about static html page scraping which is really basic, there are many sites are SPA or has lots of Javascript code these days, your best bet is using puppeteer etc though they're much slower.

as mentioned here apify is the scrapy-in-python version with javascript.