[drdaeman]

I strongly recommend investing in some lights-out management (IPMI, KVM or alike) solution that doesn’t depend on any OS peculiarities.

Configuration switching and rollback mechanisms aren’t exactly reliable with trickier setups, as it doesn’t account for any ephemeral state (like what’s actually in the routing tables), and that stuff cannot be always preemptively declared upfront. I’m afraid that despite a lot of efforts, the only truly reliable method to ensure system will come back is still to deploy-and-reboot.

[VTimofeenko]

True, it's not a full replacement for ipmi-ish tech. There are ways the deployed waiter can fail before it times out and triggers the rollback. Deploy often enough and you will hit all wonderful edge cases. I treat it as a first line of defense that saves me time on scooting the chair over to me server cabinet and yanking the cords.

WRT ephemeral state -- NixOS allows to minimize this. Coupled with impermanence, all non-declared and non-externally-retrievable state is wiped away upon reboot. And if it's not declative and not retrievable, I just don't use it. Homelab allows for a lot of choice in that regard