Hacker News new | ask | show | jobs
by JayGuerette 1340 days ago
pv is a great tool. One of it's lesser known features is throttling; transfer a file without dominating your bandwidth:

pv -L 200K < bigfile.iso | ssh somehost 'cat > bigfile.iso'

Complete with a progress bar, speed, and ETA.
2 comments
smcl 1340 days ago
Oh damn that's neat I never thought to use `ssh` directly when transferring a file, I always used `scp bigfile.iso name@server.org:path/in/destination`
link
fbergen 1340 days ago
Also see `scp -l 200 bigfile.iso name@server.org:path/in/destination`

from man page:

-l limit

Limits the used bandwidth, specified in Kbit/s.

link
MayeulC 1340 days ago
Also you probably shouldn't use scp. rsync and sftp have mostly the same semantics.

    rsync --bwlimit=2OOK bigfile.iso name@server.org:path/in/destination
    sftp -l 200 bigfile.iso name@server.org:path/in/destination
Although it seems that scp is becoming a wrapper around sftp these days:

https://www.redhat.com/en/blog/openssh-scp-deprecation-rhel-...

https://news.ycombinator.com/item?id=25005567

link
tyingq 1340 days ago
A similar trick that's nice is piping tar through ssh. Handy if you don't have rsync or something better around. Even handy for one file, since it preserves permissions, etc.

tar -cf - some/dir | ssh remote 'cd /place/to/go && tar -xvf -'

link
Twirrim 1339 days ago
I love this trick. I was dealing with some old solaris boxes something like 15 years ago when I learned you could do this. I couldn't rsync, and had started off SCP'ing hundreds of thousands of files across but it was going to take an insane length of time. Asked one of the other sysadmins if they knew a better way and they pointed out you can pipe stuff in to ssh for the other side too. Every now and then this technique proves useful in unexpected ways :)
link
dspillett 1339 days ago
Similarly, though useful less often these days, using -B/--buffer-size to increase the amount that it can buffer. If reading data from traditional hard drives, piping that data through some process, and writing the result back to the same drives, this option can increase throughput significantly by reducing head movements. It can help on other storage systems too, but usually not so much so.
link