[RestartKernel]

For anyone wanting to avoid this mystery, putting the game aside, I highly recommend NixOS. It has its quirks, but the declarative configuration basically forces you to document each system, circumventing many of the issues satirised (?) with this game.

It still gets complicated if you have many interconnected devices to manage of course, but at least it limits the obscurity of the most complex devices in your network.

I'm open to alternatives though, if anyone has some to share.

[bayindirh]

I like the idea behind the Nix and NixOS, however, it works for personal systems and small servers, not the production servers you have at work, and I tend to document those as well.

I have found a couple of ways of doing documentation.

1. I document myself, with my style: I have a big Obsidian vault with work related documents, it's loosely based on Diataxis [0] method. I live document the part I'm working on once, and when I need to repeat the deed, I can update any changed parts, if any.

2. SaltStack. People doesn't prefer it much when it comes to Ansible, but pillars and recipes allows automation tons of stuff in a self documenting format. Plus, if you wish SaltStack has drift control and anchoring (like the default CFEngine behavior).

3. Terraform. Not incorporated by me, but a project uses it, and we use it in turn. What's nice about it is scaling. They have a variables file, and change the number of the server type you want to scale, and apply. It's scaled automatically. Infrastructure as code at best.

You want to installs with specific versions and packages? You can use xCAT for RedHat and Debian systems (which will be replaced by Lenovo Conflence soonish). You can re-provision a system to the exact state you want with three commands and, within 15 minutes, your system will be ready. Since it uses the repositories local to xCAT, no version drifting occurs.

I personally install Debian stable, enable security updates, and periodically make sure that things are working. Not taking much time (10 minutes per month?).

[0]: https://diataxis.fr/

[a_t48]

I used SaltStack to manage robot OSes in the past, at the urging of my boss. It was super powerful, but it felt a little too powerful, and a little too magical.

[bayindirh]

When you dive deep into the documentation, how every action is mapped to specific OS is detailed nicely.

For example, package management actions map to specialized packages per OS, and for that specific package, documentation gives how the specific package calls the binaries or use the libraries. After a certain point you start to build call trees and see everything in your mind before pressing enter.