[gego]

...the archiveteam wiki might have some approaches that might fit your use case.

Then there is always HTTrack.