Hacker News new | ask | show | jobs
by zamubafoo 1131 days ago
That's interesting! I often found myself forgetting to turn on progress flags on many data transfer jobs and the occasional data transform batch job that I looked into something like this.

I found that `iotop` is great for this kind of thing. Sure, you have to either start it before your process starts or your accumulated total is off, but usually I'm not tracking progress for files less than 1GB so being off by kilobytes is fine.

My go-to's are `sudo iotop -aoP` for general monitoring, adding the `-p` flag if it's just a specific process, or `-u` if I'm monitoring something that is possibly transient.

2 comments

One of my quick-and-dirty gotos for getting a rough idea of buffered-writes size + disk-write activity on random linux systems is: `watch -n1 grep -ie dirty -e writeback /proc/meminfo`.

You can invoke `sync` to watch the buffered-writes queue burn down when you have lots of pending writes.

see: `LESS=+/meminfo man proc` or https://github.com/torvalds/linux/blob/master/Documentation/... for more info

I also often forgot to add progress flags, but lately I don't even bother... I just start, then `progress -w` or `watch progress -w`. Works nicely.