Hacker News new | ask | show | jobs
by int0x80 2924 days ago 

    tar cf - src/ | ssh $host "tar -C /dest/dir -xf -"
1 comments
reacharavindh 2924 days ago
Do you have any idea about a crazy fast way to transfer files between two machines on the internal network?

No encryption necessary No integrity checks No resume support Just plain and as fast it could get..

I need to try the netcat version but I'm hoping someone could show me a concurrent version of it that is mad fast.

link
int0x80 2924 days ago
You can use nc (netcat). If you have a 10G ethernet you will not saturate it because you will be limited by disk IO (I get 1Gb/s disk reads for a M.2 SSD). I you can read from disk faster, netcat alone will not saturate a 10G link, probably will hit a 3Gb/s limit (depending on your hw). You will need parallel transfers (xargs, a bit of scripting etc). You can also try rsync, maybe it is good enough for you.
link
mnarayan01 2923 days ago
From your source machine:

  < /path/to/source ncat remote-host 8001
On your destination machine:

  ncat -l 8001 > /path/to/dest
Though in most situations with files of reasonable size, you're probably going to be better off running through `gzip -c`.
link