[toomuchtodo]

Try Playwright to grab the data, do us a favor and throw it into the Internet Archive if you grab it.

https://news.ycombinator.com/item?id=39442740

[rupestrecampos]

Thanks for pointing it out, will try to use it when needed. I could put it on internet archive problem is it is updated in a daily basis so uploading to there we could more likely create a temporal analysis rather than use it for real. Other problem that some datasets are kind of big, between 1Gb and 300 Gb