If there was no way to hang up the phone you wouldn't dismiss them as easily. I think this is likely a feature of ease of dismissal rather than one of remote vs local connections
There is also an easy way to not prioritize people at the office. You tell them in a polite manner that what you are doing is a bigger priority and requires your attention. Over time people learn to coordinate discussions when you are free to talk. It is part of learning to say “No”.