[willis936]

The primary measurement is called group delay, which is the derivative of phase with respect to frequency (since a linear phase is just a uniform delay). Fortunately we have studies from the 70s publicly available that establish a group delay audibility threshold.

FIRs add vanishingly small amounts of distortion. Their real drawback is in the added delay. For asynchronous music playback this is nothing to worry about, but for video or anything interactive (communications or gaming) it's going to be a real tough pill to swallow.