But if someone calls you on the phone, they are just sending their voice to you, there's no Avatar from them, except for one your software invented. The one that you're looking at is disconnected from the actual person.
They could be video-calling you or using their own pair of glasses with their own avatar. It doesn't necessarily have to be audio-only, especially when it will get easier and easier to do this.