[dTal]

You absolutely do want to do dithering. Dithering converts distortion (error correlated with the signal, which is bad) to noise (error uncorrelated with the signal, which is less bad). This means that even though the noise floor is higher, you can recover more of the original signal. There is virtually no case where that is not desirable.

You're right of course that it will compress less well, but that's to be expected because you've lost less information!

[lmm]

Or because you've added randomness?

Store the 24-bit signal, and you could do a dithered downsample to 16-bit on playback if you think that's a good idea. Wouldn't that be better all round?