[maxmcd]

I weep for this period of time where we don't have sticky disks readily available for builds. Uploading the layer cache each time is such a coarse and time-consuming way to cache things.

Maybe building from scratch all the time is a good correctness decision? Maybe stale values in disks is a tricky enough issue to want to avoid entirely?

If you keep a stack of disks around and grab a free one when the job starts you'd end up with good speedup a lot of the time. If cost is an issue you can expire them quickly. I regularly see CI jobs spending >50% of their time downloading the same things, or compiling the same things, over and over. How many times have I triggered an action that compiled the exact same sqlite source code? Tens of thousands?

Maybe this is fine, I dunno.

[withinboredom]

Interesting.

I remember working on a project where the first clean build would always fail, and only incremental builds could succeed. I was a junior at the time, so this was 15-20 years ago. I remember spending some time trying to get it to succeed from a clean build and my lead pulling me aside: he said it was an easy fix, but if we fixed it, the ops guys would insist on building from scratch for every build. So please, stop.

Personally, unless you have an exotic build env, it’s usually faster and easier to simply build in the runner. If you need a docker image at the end, build a dockerfile that simply copies the artifacts from disk.

[adityamaru]

This is exactly the sort of insight that led us to work on Blacksmith. Since we own the hardware we run CI jobs on there are some exciting things we can do to make these "sticky disks" work the way you describe it. Stay tuned!

[parentheses]

I agree. The notion that everything must be docker is nice in principle but requires a lot of performance optimization work early on. Earlier than one would need with "sticky disks" as you called them.

[kylegalbraith]

Couldn't agree more. Somewhere, we lost the concept of disks in CI unless you run it yourself, and a lot of build tools could benefit from having them.

We came to the same conclusion and built Depot around this exact workflow for the Docker image build problem. We're now bringing that same tech into GitHub Actions workflows.

[aayushshah15]

A subtle challenge with "sticky disks" is that it requires your workflow steps to be idempotent beyond the point of "resumption", which can be tricky in a lot of cases.