[crtasm]

I believe you can deny the request for contacts list access. You can certainly add a Signal contact within the app by typing in their phone number.

I realise this doesn't solve your issue with having to share your phone number with them, or people with your number seeing you use Signal, but if you want anon communication XMPP+OTR+burner accounts is still the way to go, AFAIK.