That doesn't work as well because borg needs fast local access to the files in order to do deduplication etc. If you use SSHFS, it's going to be much slower, IIRC.
You're right. It does have an impact on performance. I haven't done a comparison, but the performance is still acceptable in my experience. It depends on the use case, but the local cache that borg maintains helps a lot in simple cases, because any unchanged files simply get skipped.