[fargle]

Toward the end of the article they use chroot to run an entire rootfs as sort of a user-level system emulation.

The next step is to do the same thing except using containers/namespaces. I was able to run a Yocto rootfs build for ARM completely, including init, and IIRC networking, using LXC and binfmt_misc. A very handy technique for testing and it does run much faster than full-system emulation.

[jraph]

Until last week, I had a full Debian amd64 systemd container to run invidious on an ARM64 machine (a ROCKpro64) running Armbian, since crystal (the language in which Invidious is written) does not have arm64 Debian packages. There's nothing special to make this work, regular commands work, for instance just running systemd-nspawn -b in an arm64 root folder works.

Now I found a way to get arm64 crystal binaries. I got rid of the container, but Invidious still cross compiles to amd64, so qemu is still used to run an amd64 build Invidious of transparently.

binfmt and qemu-user do wonders. It works well. One could use box64 [1] instead of qemu and it should provide better performance because it uses the native versions of some well known libraries (including libc6) instead of emulating them, but I failed to compile box64 this weekend so I stayed with qemu.

[1] https://box86.org/ | https://github.com/ptitSeb/box64/

[AshamedCaptain]

You don't even need containers. These techniques (a chroot + static qemu-user with binfmt) were used many years ago to cross-build (and test) entire distributions for exotic architectures.

I remember http://scratchbox.org/ would allow you to replace some components (e.g. gcc) with their native versions so as to speedup them. It is all hopelessly broken now.

[fargle]

Certainly! It's just that you might want separate mount, network, etc. namespaces. Hence: "the next step"

With a little work you should be able to do this as unprivileged/rootless as well.

[seba_dos1]

I'm still using Scratchbox (inside Docker) to build games for Maemo 5 (Nokia N900), as that's one of my engine's supported platforms ;)

[Twirrim]

I used this approach to start iterating on some Arm builds before I got access to any of the Arm servers they were introducing at work (Oracle Cloud Infrastructure).

I'd started out using a full emulation VM, and it was alright, but the cost of emulation was crippling for parts of the build process. IIRC one part of the build process was pulling in python libraries that didn't have arm wheels, and that took a bit of work to compile even on native architecture. Add in the overhead of full system emulation and it really hurt the iteration process. Especially as I worked my way from "Finally got it to build!" through to "Got the build repeatable from scratch!"

The binfmt / container approach dramatically reduced the amount of emulation being done, resulting in phenomenally faster build times.

Then I finally got access to an actual Arm instance and the entire process took even a fraction of that time.

[jakogut]

> it does run much faster than full-system emulation.

The opposite is true if you're virtualizing the same architecture as the host with hardware virtualization (KVM) enabled. Counter-intuitively, user emulation is much slower than full-system emulation in this specific case.

[spyremeown]

>I was able to run a Yocto rootfs build for ARM completely, including init, and IIRC networking, using LXC and binfmt_misc

I'd love to try that! Any pointers? :)

[fargle]

very, very, roughly:

- build a rootfs using the poky reference distro (but do it for your arm target).

- https://docs.yoctoproject.org/

- https://docs.yoctoproject.org/brief-yoctoprojectqs/index.htm...

- you'll need to make or get a layer for your machine type. for example, for rpi you'll want: https://github.com/agherzan/meta-raspberrypi

    bitbake core-image-minimal (or whatever the machine layer wants you to do)

- find the unpacked root image (or unpack the final image). should be like build/tmp/work/<machine>/<blah>-image/1.0-r0/rootfs

- run (note: in the past I used LXC, but lets try podman today)

    apt install qemu-user-static podman
    podman run -it --rootfs <rootfsloc> /sbin/init