[kbolino]

The bigger problem IMO is not that things break, it's that if you depend on one LTS release too heavily, and you wait too long to migrate from one LTS to another, everything breaks all at once.

What should be a gradual migration as new things develop turns into a singular nightmare.

[marcosdumay]

What are you depending on the OS that isn't extremely backwards compatible?

Once in a decade you get something like a breaking upgrade of nginx, or the glibc debacle of 2003. That may take a person-week to fix[1], what can hardly be called "herculean".

1 - If you go with 1 person * 1 week, if you try to go with 7 people * 1 day, it will suddenly cost 7 person-weeks. But the only way upgrading is such a hurry is if you borked a lot of things prior to it.

[kbolino]

Off the top of my head, some of the things that have broken at an LTS transition that I've been involved with are out-of-tree kernel module builds, C code using OpenSSL, Puppet config, Salt config, RPM specfiles, Python code, Perl code, Apache configs, shell scripts, Java code, bootloader configs, bootstrap scripts, and init scripts/configs (esp. sysvinit to systemd). Any one of these things is not a problem in isolation, the problem is due to having to fix all of them all at once. Too much complexity put into any one of them (often arising from external requirements or rushed implementations) also makes migrating harder. Waiting until the 11th hour on the EOL clock just adds to the stress of the process.

Many of my bad experiences were because of corporate policies and lack of proper prioritization at levels above system administration. However, the sysadmin does have some choice in the matter, especially when greenfielding. You can turn stability into a vice if you're not careful.