[rootnod3]

I used to love both, Kubernetes and Nix. But after a few years of using both I felt like the abstraction levels are a bit too deep.

Sure, it's easy to stand up a mail server in NixOS, or to just use docker/kubernetes to deploy stuff. But after a few years it felt like I don't have a single understanding of the stack. When shit hits the fan, it makes it very difficult to troubleshoot.

I am now back on running my servers on FreeBSD/OpenBSD and jails or VMM respectively. And also dumbing the stack down to just "run it in a jail, but set it up manually".

The only outlier is Immich. For some reason they only officially support the docker images but not a single clear instruction on how to set it up manually. Sure, I could look at the Dockerfiles, but many of the scripts also expect docker to be present.

And now that FreeBSD also has reproducible builds, it took one more stone away from Nix.

[ronef]

Going to sound weird but with both my hats on I super appreciate this perspective. I can only speak to some areas of Nix and Flox obviously and I know folks are looking into doing this to your point a whole lot better. Zooming in way more into solving for us that just want to run and fix it fast when it breaks.

Also, think it's a huge ecosystem win for FreeBSD pushing on reproducibility too. I think we are trending in a direction where this just becomes a critical principle for certain stacks. (also needed when you dive into AI stacks/infra...)

[rootnod3]

Yes, but I also think that the BSDs are the last bastions you will find any AI usage in. And I for one am grateful for that.

I like it when my system comes with a complete set of manpages and good docs.

But you mentioned Flox, which I didn't even know about. First I thought that's what they renamed the Nix fork to after the schism, but now I see it's a paid product and yuck...just further deepens my believe in going more bare bones manual control, even if sometimes bothersome.

[antonvs]

Kubernetes can be a godsend at larger orgs.

We have six dev teams and are just about done with migrating to k8s. It's an immense improvement over what we had before.

It's a version of Greenspun's tenth rule: "Any sufficiently complicated distributed system contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Kubernetes."

[eep_social]

I think six dev teams is small in terms of kube. I wouldn’t be surprised if that’s close to the perfect size to move onto kube and create and adopt a standard set of platform idioms.

at orgs significantly larger than that, the kube team has to aggressively spin out platform functions that enable further layering or risk getting overwhelmed trying to support and configure kube features to cover diverse team needs (that is, storage software doesn’t have the same needs or concerns as middleware or the frontend). this incubator model isn’t easy in practice. trying to adopt kube at this scale is very challenging because it requires the kube team to spin up and out sub-teams at a very high rate or risk slowing the migration down to a crawl or outright failure and purchasing e.g. off the shelf AWS because teams need to offboard their previous platform.

[antonvs]

> I think six dev teams is small in terms of kube.

I don't doubt it. By "larger" I just meant larger than something like "running my servers on FreeBSD/OpenBSD and jails or VMM respectively" above, which sounds like a one-person operation.

> I wouldn’t be surprised if that’s close to the perfect size to move onto kube and create and adopt a standard set of platform idioms.

My previous position was at a company about 5x the size, with many loosely related enterprise and government products that they sold into markets in at least 20 countries. They also used k8s quite effectively.

But, I think the key is that you mentioned "the kube team". Having a single team responsible for everything k8s-related at a large org is likely to make it difficult to be effective.

For supporting individual dev teams I think you need people on those teams who have at least some of the necessary knowledge, so they're not entirely dependent on a central team. Even a watered-down version of what devops was supposed to be about is better than nothing.

[throwaway838112]

No, most large orgs do not need it.

[antonvs]

Completely unsupported assertions are the least interesting kinds of comment anyone can post.

Besides, this particular comment would need to explain why it’s likely that its unsupported opinion is correct, rather than all the counterexamples that exist in the actual highly competitive industry that’s heavily using this system.

There’s a fundamental conflict there that doesn’t work in favor of your inchoate opinion. The most likely conclusion is that there are factors at work here that you don’t understand.