I found out that if the phone is too far away, the voice would flagged as ‚background chatting‘ and therefore muted.
Once I connected a headset those issues went away.