Signal does not has to access to contacts. It does asks for contact access permission, to show in the app the names that you have set for your contacts. But you can just answer no and everything works.
On the contrary, if you answer the same to WhatsApp, it plain refuses to work. But it actually created an account on their servers, and from that on you appear on your contacts who do use WhatsApp as another user of WhatsApp, which invites them to write to you there although you cannot receive their messages. To fix this, you have to find the option in WhatsApp to delete your account.
I don't want to speak for the parent commenter, but I think the concern is that the local app could be exfiltrating the contact list (and then by the exact same logic, message content as well) in some side channel unrelated to anything seen in the published source code, unless (a) the user builds the apk from published source code themselves, or (b) if there's some way to prove that the apk received via the Play Store is identical to one built from that source code.
Is (b) achievable by all users who have this concern?
Signal isn't a company. They're a non-profit. In addition, as others have mentioned, Signal works without giving them permissions to read your contacts.
On the contrary, if you answer the same to WhatsApp, it plain refuses to work. But it actually created an account on their servers, and from that on you appear on your contacts who do use WhatsApp as another user of WhatsApp, which invites them to write to you there although you cannot receive their messages. To fix this, you have to find the option in WhatsApp to delete your account.
Tally:
Signal 1 WhatsApp 0