[bosky101]

1) compare it with httrack website copier

2) maybe your scraping is synchronous, without perhaps any level of parallelism at the same depth

3) use your code or a sitemap to get all the URLs into a txt, then loop through with bash/curl maybe