Why not just mount the code in the container instead of copying it over? Rust deals very cleanly with build artefacts from different architectures so there's no risk of corruption.
This is a good idea but there is tension between this straightforward approach and the fact that, on macos, docker is running in a VM so it's inherently slower.
The virtualized storage layer could actually be a bottleneck here. Though I think they've improved the performance of it to the point where it might no longer be a bottleneck but I'm not sure.