[EtDybNuvCu]

Have you tried Nix? It's like Bazel, but without incremental builds. The boons are pretty great.

I'm extremely disappointed, as a career SRE, to see "hope is not a strategy" trampled in the mud. I understand that perhaps the author has succumbed to epistemic helplessness and given up on having computers work without optimism or belief, but computers do not operate on any ruleset which includes hope, and we should not encourage magical thinking.

[danmux]

Not trampling, just holding at arms length. Recognising the reality of the economics of most projects. Unless you live in a completely deterministic world, devoid of human fallibility, or perhaps omnipotent, or simply have unlimited time or resource, at some point you are going to have to admit you just don't know, and you are managing the percentages. Delusional overconfidence is more close to magical thinking than recognising the reality that complex systems will fail in surprising ways, and that it costs ever increasing resource to reduce the risks with ever diminishing returns, until you are forced to stop. The ruleset the computer abides by probably represents a fraction of the factors affecting success. If you feel you have never got to a point where there is an element of faith involved in your choice, then Im envious.

[EtDybNuvCu]

Thank you for your thoughtful reply; you deserve a serious answer.

I accept that, occasionally, my systems will fail. My data will be lost. No recovery will be possible, and the damage will be permanent and lasting. I do not hope to succeed in those times, but expect to be scarred. I will fail.

Given that I will fail, I'd like to understand how I fail. I'd like to understand why I fail. I'd like to measure how often I fail, how short I fall of success, and the root causes of my failure. I'd like to know when failure is about to happen or is happening.

By writing down what I do, I know how to fail. I can write down what I do when I don't fail, too. I can write while I do, and I can read what I wrote to do it again. I can let somebody else read and do what I have done.

I can expect to fail sometimes. I can expect to not fail sometimes. I expect failures based on causes, not based on self-blame. I expect to not fail most of the time, and only fail at certain times when something has caused me to fail.

I can fail less in the future. My actions today can change how I fail in the future. I can plan to fail, or intentionally fail, or sometimes fail less. I act intentionally.

I haven't failed in a while. The last time I failed, I looked at why I failed and I did what was necessary to try to recover and fail less.

This is how SRE works. This isn't overconfidence; this is fault-tolerance. It's not easy, but it works.

To respond to your point about faith, I have plenty of faith, just not hope that my faith will be able to prevent me from failing.

And finally, try Nix sometime. It's pretty cool.

[danmux]

Good points well made. I hope I'll be able to prioritise trying Nix at some point.

[ris]

> Have you tried Nix? It's like Bazel, but without incremental builds.

Well, depending on how granular you expect your incremental builds to be, it can indeed have "incremental builds". Any build step that produces a derivation that ends up in the nix store will be lazily evaluated and not regenerated if its input parameters haven't changed. You could go down the route of outputting each compiled object file as a derivation...

[benley]

> You could go down the route of outputting each compiled object file as a derivation...

You could, but I believe you would find that Nix becomes pretty inefficient in that scenario. There is a surprising amount of overhead involved in setup/teardown of network namespaces and the other various sandbox features, and that cost is incurred for each individual derivation. It's a reasonable tradeoff for Nix when used as a package-level build sandbox, but (assuming my understanding is correct and still current) for Nix to work well as a file-level incremental build system it would require some strategic changes.

Here's the github issue with a bunch of related discussion and details: https://github.com/NixOS/nix/issues/179

[ris]

Yes it's all interesting stuff.

But, one way of looking at it, that's just ("just") an implementation issue. No ideas fundamental to Nix would have to be changed to find a faster way of implementing derivation isolation, hopefully any solution to problems like this could be transparent to people who had written build systems in Nix and not require significant rewrites...?

Even so, incremental builds wouldn't be an all-or-nothing scenario. A project could be divided up into a number of separately buildable partitions, each built as a separate derivation. Deciding how granular to make your incremental..ism would just be a tradeoff of overhead vs. amount of rebuilding saved.