Ok, I'm actually only 90% wrong! It turns out that there is in fact some information about the BH-NS distinction in the gravitational signal itself, but it is very weak and hasn't been detected yet. It is expected to be detected in the future though.