Yeah I didn't particularly like that about Signal. If a malignant person were trying to keep tabs on a person (victim) using Signal, they would be able to see that the victim has switched to encrypted messaging (letting the malignant person know the victim is at least partially onto them). Thinking about the use case of a person trying to avoid a hacker/stalker here for example.
If your adversary is a nation state, hiding your phone number is not an option. They already know it and might compromise a device. Perhaps another service is better or the use of a burner phone is preferred.