[adobrawy]

I previously tried to use Docker `docker/setup-qemu-action@v2` and `docker/setup-buildx-action@v2` for this purpose (see that example https://github.com/docker/build-push-action#git-context). Thanks to buildkit, platform switching works transparently. However, building on ARM via QEMU on GitHub Actions is terribly slow (something like 5 times more), which is hard to accept. Therefore, full of hope, I am waiting for GitHub Actions to make cloud runners available on ARM, because it is a blocker for the implementation of Graviton on the AWS environment for us.

For a while, the blocker in GitHub Actions for providing ARM support was that Azure doesn't have ARM support. In this way, the Azure cloud offering may determine the habits of AWS consumers.

[kylegalbraith]

This was one of the frustrations that led us to create Depot (https://depot.dev/). It's a remote Docker build service that launches Intel & Arm VMs to build native multi-platform images in parallel on native CPUs with zero emulation.

Behind the scenes we are running the same engine as Docker, BuildKit, on VMs in AWS for both architectures. We combine native architecture support with a 50GB persistent SSD cache that is automatically available across builds, so no more saving/loading cache over the network.

All those things combined makes multi-platform or Arm image builds in GitHub Actions ~15x faster using Depot (example: https://depot.dev/benchmark/temporal). And it's a one line change, swap `docker build` for `depot build` or use our `depot/build-push-action` instead of `docker/build-push-action`.

[adobrawy]

Are you SOC2 compliant? Can an image built in this way be loaded back (`load: true`) to CI without OCI registry?

[kylegalbraith]

We aren't SOC2 compliant yet, but we are planning to do that this year. Yup, you can pass `depot build --load` (the same parameters you use with buildx) and we will return the built image to you. Or you can use `--push` and we will push it up to your registry.

Here is my direct email if anyone wants to chat/learn/share more about this problem space, kyle [at] depot.dev.

[JacobCarlborg]

Yeah, I've been using `docker/setup-qemu-action` as well to run on non-x86-64 architectures for Linux [1]. But since it's Docker it's Linux only and doesn't support other operating systems.

GitHub has Apple Silicon on the roadmap, but IIRC it was at the end of this year. They already support Apple Silicon for self hosted runners [2].

[1] https://github.com/jacob-carlborg/lime/blob/f4d9c8c4265b61b2...

[2] https://github.blog/changelog/2022-08-09-github-actions-self...

[saurik]

Why do you need to build on ARM to target ARM? I support ARM and I just build on x86_64.

[adobrawy]

I need to build images for an application in Python. Cross-build Docker images? I haven't heard of anything like that. I need to build images for an application in Python. Cross-build Docker images? I haven't heard of anything like that.

Did you mean cross-compilations for compiled languages? It doesn't fit Python.

[saurik]

I am sorry but I am having a difficult time understanding your comment. If you are coding in Python, why does the underlying architecture matter at all?

[dbingham]

The architecture of the docker image matters. If you build a docker image on an ARM machine (like a m1 or m2 Mac) you can't run it in x86 architecture (like a T3 or M5 AWS instance). If you build on x86 architecture, you can't run it on ARM architecture.

That is, unless you use Docker's cross platform build capability (buildx). Which was still considered experimental the last time I looked at it (about a year ago).

[Arnavion]

You can build other-arch images with regular `build`. You'll of course need QEMU hooked up through binfmt to be able to execute `RUN` steps while building the image, but you can do that yourself without involving `buildx`.

[dbingham]

Neat! I need to read up on QEMU. I haven't looked into cross arch images in a while.

[hk1337]

Some packages it doesn’t matter the architecture but others it actually does. Pandas, for example.

[Arnavion]

At $dayjob we build packages of our software for a bunch of Linux distros, which means the software has to be compiled individually for each distro to get external dependencies right. Some of those distros like the RHEL 7 family don't support cross-compilation, so we run them in QEMU.

[saurik]

You just need a sysroot for that distribution... you actually don't want to compile ON that distribution as it makes having a consistent and modern/working toolchain a lot more difficult.

Notably, I compile for CentOS 6 (ancient, right?) on whatever the latest version of Ubuntu is--using the most recent versions of clang and rust and whatever--and simply build my sysroot using this trivial script.

https://github.com/OrchidTechnologies/orchid/blob/6958658c25...

(You can do this in some sense even easier using Docker--though in other senses it is a lot more complex as now you need docker--if you are into that sort of thing. You don't run the compiler in docker: you just install the dependency packages and then docker export the filesystem as your sysroot.)

Using this script and clang I can actually compile to target CentOS 6 on macOS (and I get the exact same binary if I use a consistent version of clang; I actually have a GitHub action that verifies that I reproduce the same binary compiling on both macOS and Ubuntu).

[Arnavion]

>You just need a sysroot for that distribution... you actually don't want to compile ON that distribution

We compile in a Docker container of the distro. It's the same concept as having a sysroot.

>as it makes having a consistent toolchain a lot more difficult.

>using the most recent versions of clang and rust and whatever

The external dependencies I mentioned are libraries like glibc or openssl, where you want to use the distro version. Similarly the packages should be consistent with what the users of the distro would build themselves if they used rpmbuild / debbuild / whatever. The toolchain not being consistent across distros is the point.

Also, none of this is relevant to cross-compiling for ARM. As I said, RHEL 7 doesn't have a cross compiler - specifically it has gcc but not glibc, because the cross compiler is only meant for compiling the kernel.

[saurik]

> We compile in a Docker container of the distro. It's the same concept as having a sysroot.

It isn't the same concept as using a sysroot, as you are now using the compiler from that potentially-ancient different distribution. I see people constantly twisting themselves into knots being like "CentOS has some broken/limited build of gcc/clang so I can't use X" and it is like "if there is one thing you as a developer control it is the compiler".

> The external dependencies I mentioned are libraries like glibc or openssl, where you want to use the distro version.

I understood this: that's what you put in your sysroot (and is easily seen in the example script I showed for CentOS 6; my scripts for building Ubuntu sysroots are much more complicated as I am trying to avoid nesting docker and so now have some crazy fallback involving debootstrap and proot).

> Similarly the packages should be consistent with what the users of the distro would build themselves if they used rpmbuild / debbuild / whatever.

Ok, this is where we differ, then: I am not trying to simultaneously support users building on CentOS using "rpmbuild". I want to be able to use the latest build of C++ and Rust and Python and essentially have a "modern"/easy development stack but "target"/support other distributions so they can install my software as packages.

Given the goal of being able to provide source code that can itself be compiled on these random distributions I see the problem. I personally feel like it would still be better to work around that by requesting (which can itself be automated by your build) that the user compile their own toolchain that has more functionality, though.

[Arnavion]

>it is like "if there is one thing you as a developer control it is the compiler".

>I understood this: that's what you put in your sysroot

You're advocating mixing compilers and libraries that are not from the same distro release. While it may work for you it's generally a recipe for disaster.

The correct and least-mental-overhead way of building packages for $distro is to build on $distro. The fact that users of $distro can then build your package themselves using the standard distro method is an extra benefit.