Hacker News new | ask | show | jobs
by yjftsjthsd-h 191 days ago
If the difference is <1 minute vs >30 minutes, containers (per se) are not the problem. If I was guessing blindly, it sounds like you're not caching/reusing layers, effectively throwing out a super easy way to cache intermediate artifacts and trashing performance for no good reason. And in fact, this is also a place where I think docker - when used correctly - is quite good, because if you (re)use layers sensibly it's trivial to get build caching without having to figure out a per-(language|build system|project) caching system.
1 comments

I'm exaggerating somewhat. But I'm familiar with Docker's multi-stage builds and how to attempt to optimize cache layers. The first problem that you run into, with ephemeral runners, is where the Docker cache is supposed to be downloaded from, and it's often not faster at all compared to re-downloading artifacts (network calls are network calls, and files are files after all). This is fundamentally different from per-language caching systems where libraries are known to be a dumb mirror of upstream, often hash-addressed for modern packaging, and thus are safe to share between builds, which means that it is safe to keep them on the CI runner and not be forced to download the cache for a build before starting it.

> without having to figure out a per-language caching systems

But most companies, even large ones, tend to standardize on no more than a handful of languages. Typescript, Python, Go, Java... I don't need something that'll handle caching for PHP or Erlang or Nix (not that you can really work easily with Nix inside a container...) or OCaml or Haskell... Yeah I do think there's a lot of room for companies to say, this is the standardized supported stack, and we put in some time to optimize the shit out of it because the DX dividends are incredible.

I really don't see how that's different at all, certainly not fundamentally. You can download flat files over the network, and you can download OCI image layers over the network. I'm pretty sure those image layers are hash-addressed and safe to share between builds, too, and you should make every effort to keep them on the CI runner and reuse them.