I worked with cross-compiling containers. The compiling times were... bad. Our x86 build took like, two minutes (this was a very small and lean C++ application). The arm32v7 ones took upwards of 30 minutes.
Works the other way around as well: Compile platform independent code (such as Java) on --platform=$BUILDPLATFORM in a build stage and then copy into containers that are --platform=$TARGETPLATFORM. That way your build only runs once, natively, but you can produce the correct runtime containers for each architecture rather quickly.