virustotal.com allows you to upload files to scan with a whole range of anti-virus programs. Before uploading, it will calculate the hash of your file client-side to see if the file should be uploaded or if a previously uploaded (by someone else) file with same hash should be re-scanned with newer versions of the anti-virus.
I don't know which hashing algorithm they use but just as example of a situation where whitelist is not used.
Yes, I think that's what the author was alluding to here, although I'm not sure:
The approach may work with traditional AV software too as
many of these also use fingerprinting (not necessarily MD5)
to avoid wasting resources on scanning the same files over
and over (although the RC4 encryption results in VT 0/57
anyway…).
I don't know which hashing algorithm they use but just as example of a situation where whitelist is not used.