VC is quite chatty and assumes that filesystem operations have a negligible cost. Before I disabled it, VC was adding >1 second to every find-file operation over Tramp.
I also recommend using the direct-async-process connection property[2], which significantly decreases the latency of async process creation.
I'm seeing ~ 7 seconds of delay compared to the excrutiating ~20 seconds I was seeing before.