Coincidentally I set this up last night under UTM (https://docs.getutm.app/advanced/rosetta/) and it worked astonishingly well. Outside of Rosetta, the native virt seemed a bit slower than whatever UTM does normally, but perfectly usable on my M1 MBP.
If you're saying the rest of the aarch64 VM got slower after enabling Rosetta 2, that's unfortunately expected. Under Linux, TSO is enabled for everything, even aarch64 processes when Rosetta is active. On macOS, TSO is selectively enabled only for the x64 processes and leaves everything else alone.