Hacker News new | ask | show | jobs
by KaiserPro 4292 days ago
WE use 10 rsyncs in parallel to copy .5pb in less than 3 days.

CPU is cheap.
1 comments
spydum 4291 days ago
Have done the same, found that rsync seems to not natively parallelize itself, so spread across 20 cores, it really screamed.
link
rimantas 4291 days ago
Damn, I could use some advice there. Right now I have a system which uses rsync to backup some files. It takes about 8 hours to go over 7+ million files (~3.5TB). I'd love to speed up this. I should mention that copy is done over curlftpfs :(.
link