Depending on the job, team building and social trust are important aspects of a well functioning team. If you truly do work alone, fine. But most people don't.
I don't work alone. Most of what I do involves getting other people to change what they're doing. It's high-interaction work, for the most part. And it works just fine remotely.