|
|
|
|
|
|
by _a9
1129 days ago
|
|
|
I use browsertrix-crawler[0] for crawling and it does well on JS heavy sites since it uses a real browser to request pages. Even has options to load browser profiles so you can crawl while being authenticated on sites. [0] https://github.com/webrecorder/browsertrix-crawler |
|
|