If you have unattended-upgrade and automatic reboot in the cloud to benefit from security updates for long-lived instances, then you better make sure to have a tty console attached to it. You are treating it like a physical machine, you must have the same tooling around.
Not necessarily. With good timing and some luck, you can connect the serial/"recovery" console before GRUB's timeout ends and either change the running kernel or add the `systemd.mask=docker.service` boot parameter to prevent Docker from starting.
Nope. Back before VMs were thing it was common to do "lights out" style remote management via a console server. That console server would then have a serial connection (the old 9 pin d-sub plug[1]) to your individual physical servers. You could then connect to your remote servers local TTY via the console server a little like jumping to remote servers via an SSH bastion. However it did sometimes require a little bit of prior configuration, depending on your distro[2].
This wasn't just limited to Linux either. It was a common UNIX trick :)
This is a bit of a lost art these days though. iLo, IPMI have replaced the need for serial. Then virtualisation and, to a lesser extent, containerisation have lowered the bar even further plus also moving the industry towards more ephemeral systems that can be destroyed and rebuilt automatically rather than the old habits of nursing failed hosts back to health.
Most smaller teams usually don’t prioritize physical access — they usually only need it for one-off events. While this would be a one-off event, it would be one that affects many servers.
That is really not my experience at all. Every professional smaller team I worked with "usually" had this figured out and set up.
In times of home office, no one wants to be at the office for just pressing a single button on some server.
My experiences for ops is all pre-2012 and with teams numbering less than 3 for the whole org. So I’m sure things have changed or gotten cheaper? I can’t see a team of 3-4 having the budget to get something that allows them to be “lazy”, especially when that budget can go towards something useful. But I guess the pandemic probably changed things there?
Serial connections will only cost you a Raspberry Pi (there's probably some really cheap console servers on eBay too).
I don't think the issue is so much cost but more this kind of systems administration is becoming a forgotten art because 99% of the time modern tooling removes the need for it. So younger sysadmins are never taught how to do these kinds things. However when I started out, I worked in a few small companies that had their physical hosts connected to a console server (which was a Cisco device like a network switch) via serial cables and you'd then connect to that console server remotely.
If you can afford to have something down for an extended period then fine. But even with a small team some services are built such that certain device outages cannot be tolerated, at least for an extended period.
So out-of-band/console servers or whatever still make a lot of sense and a relatively high priority.