Can you retest with experimental virtualization enabled? At least on my M1 this boosted performance a LOT (in addition to using gRPC FUSE mounts, as a other commenter also posted)
I’m going to try this later. I noticed the slowdown switching from a relatively weak Linux server to my M1 Macbook and had already accepted the disappointment of the M1 not being so great after all.