[tiborsaas]

I've just discovered Headless Chrome crawler and it works pretty well. Not sure how well it will scale, but I'll index a few hundred sites only.

https://github.com/yujiosaka/headless-chrome-crawler

[mnmkng]

I have not tried the Headless Chrome Crawler personally, but try the Apify SDK out https://github.com/apify/apify-js if the Headless Chrome crawler does not scale well enough. We use it to scrape billions of pages every month.