[kccqzy]

> But the Dockerfile has continued because of its flexibility.

The flip side is that the world still hasn’t settled on a language-neutral build tool that works for all languages. Therefore we resort to running arbitrary commands to invoke language-specific package managers. In an alternate timeline where everyone uses Nix or Bazel or some such, docker build would be laughed out of the window.

[muvlon]

As a Nix evangelist, I have to say: Nix is really not capable of replacing languag-specific package managers.

> running arbitrary commands to invoke language-specific package managers.

This is exactly what we do in Nix. You see this everywhere in nixpkgs.

What sets apart Nix from docker is not that it works well at a finer granularity, i.e. source-file-level, but that it has real hermeticity and thus reliable caching. That is, we also run arbitrary commands, but they don't get to talk to the internet and thus don't get to e.g. `apt update`.

In a Dockerfile, you can `apt update` all you want, and this makes the build layer cache a very leaky abstraction. This is merely an annoyance when working on an individual container build but would be a complete dealbreaker at linux-distro-scale, which is what Nix operates at.

[kccqzy]

Fundamentally speaking, the key point is really just hermeticity and reliable caching. Running arbitrary commands is never the problem anyways. What makes gcc a blessed command but the compiler for my own language an "arbitrary" command anyways?

And in languages with insufficient abstraction power like C and Go, you often need to invoke a code generation tool to generate the sources; that's an extremely arbitrary command. These are just non-problems if you have hermetic builds and reliable caching.

[xhcuvuvyc]

I mean, I guess at a theoretical level. In practice, it's just not a large problem.

[poly2it]

Well, arbitrary granularity is possible with Nix, but the build systems of today simply do not utilise it. I've for example written an experimental C build system for Nix which handles all compiler orchestration and it works great, you get minimal recompilations and free distributed builds. It would be awesome if something like this was actually available for major languages (Rust?). Let me know if you're working on or have seen anything like this!

[nh2]

A problem with that is that Nix is slow.

On my nixos-rebuild, building a simple config file for in /etc takes much longer than a common gcc invocation to compile a C file. I suspect that is due to something in Nix's Linux sandbox setup being slow, or at least I remember some issue discussions around that; I think the worst part of that got improved but it's still quite slow today.

Because of that, it's much faster to do N build steps inside 1 nix build sandbox, than the other way around.

Another issue is that some programming languages have build systems that are better than the "oneshot" compilation used by most programming languages (one compiler invocation per file producing one object file, e.g. ` gcc x.c x.o`). For example, Haskell has `ghc --make` which compiles the whole project in one compiler invocation, with very smart recompilation avoidance (pet-function, comment changes don't affect compilation, etc) and avoidance of repeat steps (e.g. parsing/deserialising inputs to a module's compilation only once and keeping them in memory) and compiler startup cost.

Combining that with per-file general-purpose hermetic build systems is difficult and currently not implemented anywhere as far as I can tell.

To get something similar with Nix, the language-specific build system would have to invoke Nix in a very fine-grained way, e.g. to get "avoidance of codegen if only a comment changed", Nix would have to be invoked at each of the parser/desugar/codegen parts of the compiler.

I guess a solution to that is to make the oneshot mode much faster by better serialisation caching.

[poly2it]

What if you set up a sandbox pool? Maybe I'm rambling, I haven't read much Nix source code, but that should allow for only a couple of milliseconds of latency on these types of builds. I have considered forking Nix to make this work, but in my testing with my experimental build system, I never experienced much latency in builds. The trick to reduce latency in development builds is to forcibly disable the network lookups which normally happen before Nix starts building a derivation:

    preferLocalBuild = true;
    allowSubstitutes = false;

Set these in each derivation. The most impactful thing you could do in a Nix fork according to my testing in this case is to build derivations preemptively while you are fetching substitutes and caches simultaneously, instead of doing it in order.

If you are interested in seeing my experiment, it's open on your favourite forge:

https://github.com/poly2it/kein

[vatsachak]

crane

https://crane.dev/getting-started.html

[poly2it]

I use crane, but it does not have arbitrary granularity. The end goal would be something which handled all builds in Nix.

[brightball]

Reminds me of the “Electric cars in reverse” video where the guy envisions a world where all vehicles are electric and tries to make the argument for gas engines.

[anal_reactor]

Link?

[fittom]

try searching for 'Rory Sutherland: What If Petrol Cars Were Invented In 2025'

[brightbeige]

Actual article was in the evening standard, but like all things Rory Sutherland, it’s worth to watch him tell the story: https://youtu.be/OTOKws45kCo?si=jbTdx3YCGkZv3Akb

For those who want more of him, check out his classic TED talk from decades ago: “Lessons from an ad man”

https://www.ted.com/talks/rory_sutherland_life_lessons_from_...

[_the_inflator]

There is some truth to it, however in production it is simple: There is a working deployment or not.

Therefore I would rephrase your remarks as upside: let others continue scratch their head while others deploy working code to PROD.

I am glad there is a solution like Docker - with all it flaws. Nothing is flawless, there is always just yet another sub-optimal solution weighting out the others by a large margin.

[kccqzy]

Popularity of a technology usually isn’t perfectly correlated with how good it is.

> let others continue scratch their head while others deploy working code to PROD.

You make it sound like when docker build arrived on the scene, a cross-language hermetic build tool was still a research project. That’s just untrue.

[whurley23]

This calls for https://xkcd.com/927/.