[indrora]

I wrote a tool a short bit ago to handle massive parallel hashing. Originally, it was a testbed for a project I built to make multiprocessing easier in Go.

https://github.com/indrora/hyperhash

In it, I learned that there's some interesting quirks about the way the traditional shaXsum tools work, specifically that they focus on a set of inputs from the shell command line, which means that you can exhaust the shell command line length fairly easily (try hashing every file in the Linux source tree, for instance; zsh, bash, and fish will all eventually tell you "Fuck off, there's too many characters in argv")

I also discovered that Go's handling of file handles varies drastically based on OS. Windows, for instance, has no problem with you taking a file handle to every single file on a disk. Linux will get upset but eventually capitulate, macos will stomp its feet and inform you that you are a bad child and kill you off.

[optionalsquid]

> In it, I learned that there's some interesting quirks about the way the traditional shaXsum tools work, specifically that they focus on a set of inputs from the shell command line, which means that you can exhaust the shell command line length fairly easily (try hashing every file in the Linux source tree, for instance; zsh, bash, and fish will all eventually tell you "Fuck off, there's too many characters in argv")

Did you try to run something like

  sha256sum **/*.*

Because if so, then you'll of course exceed the maximum number of arguments. That applies to all command-line tools that take an unbounded number of arguments.

A simple solution is to use find, which handles that for you:

  find -type f -exec sha256sum "{}" "+"  # 1.4s on my system for Linux 7.1.5

Alternatively, you can combine find with xargs (or parallel), which also handles it and optionally allows you to specify exactly how many arguments to process per call (-n), and how many commands to run in parallel (-P):

  find -type f -print0 | xargs --null -P 8 -n 512 sha256sum  # 0.3s

[lanstin]

I always have all these parallel things take a configurable number of worker go routines, then try different numbers and run btop or iostat to see how close I am to being hardware bound. Never had a machine where the limit was anywhere close to the number of open FDs allowed. I guess I haven't proven this but my idea is having the generic scheduler have to sort 60k FDs or 2M (I have a lot of dup files in my "backup everything, randomly and repeatedly, since 1998 file system) FDs is going to thrash stuff more than just having go's runtime and my own code.