[plusquamperfekt]

NSA only knows then that somebody with that phone number is using Signal ... so what?

They would find out anyway b/c your packets are sniffed on a regular basis and so they know your phone is communicating with the Signal server and maybe the packets itself are characteristic.

So in my humble opinion this criticism is irrelevant.

[xorcist]

Isn't the criticism that Signal maps your contact metadata?

An inside man is much easier for NSA than all that expensive world wide data collection. (Of course, in reality surely they do both.)