[lizknope]

In the case of a bad Linux kernel update I would just reboot and pick the previous kernel from the boot menu. By default most Linux distributions keep the last 3. I'm not an IPMI remote management expert but it may be possible to script this.

All my machines at home run Linux except for my work laptop. It is stuck in this infinite blue screen reboot loop. Because we use Bitlocker I can't even get it into safe mode or whatever to delete the bad file. I think IT will have to manually go around to literally 8,000 work laptops and fix them individually.

[t_spins]

You would "just pick the previous kernel from the boot menu". That's funny, cause in this case you could "just delete the file causing the issue." Anything can sound easy and simple if you state it that way.

How do you access the boot menu for a server running in the cloud, which you normally just SSH into (RDP in Windows' case)?

About your last paragraph: we have just started sending out the bitlocker keys to everyone so it can be done by them too. Surely not best practice, but it beats everyone having to line up at the helpdesk.

[cesarb]

> You would "just pick the previous kernel from the boot menu". That's funny, cause in this case you could "just delete the file causing the issue." Anything can sound easy and simple if you state it that way.

One small difference, is that choosing the kernel from the boot menu is done before unlocking the encrypted drive, so no recovery keys would be necessary. And yes, choosing an entry from a menu (which automatically appears when the last boot has failed) is simpler than entering recovery mode and typing a command, even without disk encryption.

A better analogue would be a bad update on a non-kernel package which is critical to the boot sequence, for instance systemd or glibc. Unless it's one of the distributions which snapshot the whole root filesystem before doing a package update.

[__MatrixMan__]

NixOS boots to a menu of system configuration revisions to chose from which includes any config change, not just kernel updates.

It's not filesystem snapshots either. It keeps track of input parameters and then "rebuilds" the system to achieve the desired state. It sounds like it would be slow, but you've still got those build outputs cached from the first time, so it's quite snappy.

If you took a bad update, and then boot to a previous revision, the bad update is still in the cache, but it's not pointed to by anything. Admittedly it takes some discipline to maintain that determinism, but it's discipline that pays off.

[rewgs]

I hate to be the guy that's like "Nix is the solution," but...Nix is the solution.

Nearly every corporate machine that needs to run Windows should run it as a VM on a NixOS base, unless there is an extremely good reason not to.

[__MatrixMan__]

Progress is slow, but eventually there will be nix on windows: https://discourse.nixos.org/t/nix-on-windows/1113/117 (fingers crossed).

I don't expect to use it much myself but I love the idea of reducing the OS to an interchangeable part. What matters is the software and its configuration. If windows won't boot for some reason, boot to the exact same environment but on a different OS, and get on with your day.

If something is broken about your environment, fix it in the code that generates that environment--not by booting into safe mode and deleting some file. Tamper with the cause, not with the effect. Cattle, not pets, etc.

This sort of thing is only possible with nix (and maybe a few others) because elsewhere "the exact same environment" is insufficiently defined, there's just not enough information to generate it in an OS-agnostic way.

[lizknope]

I can't delete a file if the machine doesn't finish booting. Unless you are suggesting removing the drive and putting it in another machine. That requires a screwdriver and 5 minutes vs. the 10 seconds to reboot and pick a different kernel.

I'm not talking about the cloud. I am talking about the physical machines sitting in front of me specifically my work laptop.

I am an integrated circuit computer chip designer, not a data center IT person. I have seen IPMI on the servers in our office. Do cloud data centers have this available to people?

I have a cheap cloud VM that I pay $3.50 a month. I normally just SSH in but if I want to install a new operating system or SSH is not responding then I log in to the web site and get a management console. I can get a terminal window and login, I can force a reboot, or I can upload an ISO image of another operating system and select that as the boot device for the next reboot and install that.

Does your cloud service not have something like this?

I don't know what our corporate IT dept wants to do. We all work from home on Friday and I can't login to check email so I'll just wait until Monday as there is nothing urgent today anyway.

[squeaky-clean]

Booting into safe mode still works to delete the bad file.

[lizknope]

The OS drive is encrypted with Bitlocker. I've seen another thread where corporate IT departments were giving out the recovery key to users. I don't need to get anything done today. I'll go into the office on Monday and see what they say.

[j33zusjuice]

Idk if this is a serious question, but you just turn on console access in the cloud provider. It’s super easy. Same concept as VMWare. It’s possible that not all cloud providers do that, I suppose.

[vel0city]

The biggest cloud providers out there (AWS, Azure, GCP) don't.

[kelsey98765431]

> How do you access the boot menu for a server running in the cloud, which you normally just SSH into (RDP in Windows' case)?

They just said IMPI.