Hacker News new | ask | show | jobs
by akx 1438 days ago
> have to follow the procedure for restoring from backup or re-mounting the root volume on an alternate house to revert the kernel version being run.

Or add `systemd.mask=docker.service` to your boot parameters to prevent Docker from starting.
1 comments
capableweb 1438 days ago
Which, if your server is stuck in a infinitive "boot -> docker starting -> container starting -> crashing kernel -> reboot" loop, you won't ever get a chance of actually adding anything to your boot parameters.
link
dspillett 1438 days ago
If you have access to the console (local physical machine, VM on a system that can expose the console, physical box that you have console access to via IPMI or other means), can you not specify that directive to be passed through via grub's interactive menu?

Failing that you could try the “single” directive and poke other configurations once booted in that mode.

A faf to be sure, but hopefully viable options (assuming the interactive menu hasn't been disabled to save a few seconds off boot time!).

link
bravetraveler 1437 days ago
Absolutely can, I'm quite surprised at the 'what do' attitude around this. It's routine -- not in all organizations to be sure, but it's a solved problem.

There are options even without out of band management. You can choose to configure your systems with PXE -- if the installation ever fails, it can boot into a recovery environment over the network.

link
jacquesm 1438 days ago
That's not correct. If you stop the boot you can add 'single' to the boot statement which will drop you in a single user shell from where you can do quite a bit of maintenance.
link
markstos 1438 days ago
AWS at least provides serial console access so have the option to access it during the boot cycle.

Alternatively, you umount the drive, attach it to another machine, chroot into it, fix grub or whatever, reverse the process and boot again. It's a few steps, but can be done in a few minutes with practice.

link
bravetraveler 1438 days ago
Out of band management is common and highly recommended
link
mrintegrity 1437 days ago
Actually networking and ssh come up for a couple of seconds before containerd triggers the kernel panic so you can fix it by doing this:

while true; do ssh <servername> sudo mv /usr/bin/containerd /usr/bin/containerd.backup ; sleep 1; done

While rebooting the system

link
gtirloni 1438 days ago
Sorry, what? You don't lose complete control of a server because it's rebooting nonstop.
link