[boronine]

I've spent days trying all of these solution at my company. All of these solutions suck, they are slow and only successful builds get their layers cached. This is a dead end. The only workable solution is to have a self-hosted runner with a big disk.

[kylegalbraith]

This is definitely a direction to try. But if its faster Docker image builds and a layer caching system that actually works, you should definitely try out Depot. We automatically persist layer cache to persistent NVMe devices and orchestrate that to be immediately available across builds.

[paholg]

If you have the ability to choose the tooling, I recommend looking into building docker images with nix.

Yes, nix is complex. But its caching story is soooo much better than docker's, and all the other docker issues just disappear.

https://nix.dev/tutorials/nixos/building-and-running-docker-...

[teaearlgraycold]

I use namespace’s action runners for this (just a customer, not affiliated in any way). They’re a company with a pretty good product stack. Although the web UI is annoyingly barebones.

[20thr]

Hi -- Namespace's CEO here; if you have a chance, please drop me a note at hugo-at-namespacelabs.com; I'd love to hear what we could be doing better in the UI, and product overall. Thank you!

Hugo @ Namespace (https://namespace.so)

[mysza]

Can you share example of github actions? When i use docker/setup-buildx-action and local runner i can't make it use the cache. I think it's the "docker-container" runner's fault

[aayushshah15]

how do you ensure isolation between runs on a self hosted runner that way?

[boronine]

What kind of isolation do you need? We are building our own code so I don't see the need for isolation beyond a clear directory.

[airspeedjangle]

Shared runner infrastructure in a big company. It's pretty common to treat these situations as multi-tenant low trust environments.

[damianh]

Plenty of marketplace actions will install things and/or mutate the runner. It's a matter of time before someone does something or there's a build that doesn't cleannup after itself (e.g. leaving test processes running) that ruins the day for everyone else.

[withinboredom]

If you are in k8s, you can use the default chart provided by GitHub and get 90% of the way there.

[damianh]

Selh-hosted runners can be ephemeral too. With such either mount the cache as a disk or bake docker layers/images into the runner image.

[remdoWater]

This requires a lot of work from a dev inf team, though. Not as straightforward for an average team.

[damianh]

I won't disagree. It should be easier imo. I guess this is why a cottage industry has sprung up addressing such e.g. https://news.ycombinator.com/item?id=39930908

[intelVISA]

Now I've seen everything...

[ownagefool]

It's actually pretty easy.

Setup GitHub app. Install the arc helm charts. Install a buildkitd statefulset.

Update parans on build to use buildkitd.

That's not to say there aren't better caching strategies, but a really basic ephemeral setup is right there.

[boundlessdreamz]

runs-ons supports custom images - https://runs-on.com/features/byoi/ and caching to S3 - https://runs-on.com/reference/caching/

I haven't used it yet but these two features make it the clear favourite for me in alternate github action runners