[obelix_]

If the site is mediawiki based check out kiwix. They have tools to download the whole site with search index. Also tools to search/reopen and render pages from the dump.

I suppose similar tools have been created by the internet archive folk.

But you are right it should be much easier to archive stuff in 2018 and it isn't esp thanks to all the JavaScript and XHR happening.

Edit: I just took a look at kiwix (been a while) they seem to also now archive stackexchange sites not just mediawiki...so looks like they have different archiving tools for different sites.