Dithering modulates the least significant bit of the audio signal, so at normal listening levels it should be inaudible. Of course, one can amplify the silence by 120dB or so and hear the noise.
I often wondered about the noise I was hearing from my Discman with headphones. You'd hear this hiss when you pressed play, even before the CD loaded, so it was probably the cheap consumer audio hardware rather than the CD format itself.
I think an additional layer of hiss would show up when it actually started playing.
Indeed, no inexpensive portable digital electronics was going to give you perceptible-noise-free 16-bit audio.
Depending on how the original audio was mastered, it could also have it's own hiss in the sound data if the CD itself. You can tell if you load the raw CD data into a wave editor and see low amplitude noise where you'd expect silence.
I think an additional layer of hiss would show up when it actually started playing.