[mezyt]

A half conversation is a lot more disruptive because your brain try to fill in the gap of information.

[cortesoft]

This comment chain is talking about people using speakerphone, though, meaning they hear both sides of the conversation

[jomohke]

In theory yes, but in practice they usually have the speaker up far higher than they are speaking themselves so we do only hear one side clearly.

I think the high distractability is a trifecta of volume, non-naturallness of the sound (compression etc: feeling out of place in the space) and this point.