[hardwaresofton]

Agreed, PXE seems ideal for provisioning things, but it's just too hard to use, especially when you're not on a network you fully control.

I just want to start the computer, and have it download an immutable OS image from somewhere I decide (and supply a checksum for, etc). I don't want to set up TFTP or any of this other stuff. It feels like I should be able to just specify an IP (let's say) a checksum (maybe supply that information to the NIC directly somehow), and be off to the races after a reboot.

[convolvatron]

replace the PXE stack with an OS installer written in UEFI. This bootload can be installed through a guest running on the host in the EFI partition, or possibly through PXE or direct UEFI http load.

this allows you intermediate the boot process without coordinating with the administrative owner of the DHCP server, and is actually less janky than PXE

[hardwaresofton]

Not to sound incredibly lazy (because I am), but is there anything off the shelf that does this? Anyone doing something similar would also be great -- I saw some UEFI-in-Rust projects recently so maybe it's not too hard to hack through myself.

That said, my original need for this was on Hetzner, and what I did instead was actually completely automate their reset setup (thankfully they have an API to that), so I have a solution, but I would much rather if I could prebake and load images much easier.

I think these days Hetzner has much more UEFI support across it's dedicated server fleet.

Side note: Every time I see tinkerbell and other relatively new PXE boot projecs and it's set of tools I feel relief, then dig in, then feel dread again around iPXE.

Side note 2: I really want to do two things:

1. Load OSes from the network easily

2. Run the OS in RAM (ECC)

I feel like it would take my server management experience to the next level -- I've spent an inordinate amount of time messing with Hetzner USB add-ons and Alpine to try to get things to work, but it wasn't reliable (it worked, but wasn't reliable).

[ahepp]

If you're familiar with the Linux boot process, you may be aware that the system often does run from RAM for a period of time, before mounting a filesystem from a block device and calling pivot_root to transfer the rootfs over.

If you want to run the OS from RAM, the absolute simplest way to do it is to simply never transfer. Building a custom initramfs has never been easier. There are tools explicitly geared towards making an initramfs, like Dracut, but also a huge ecosystem of tools for building container images which could almost certainly be scripted to spit out a cpio without a lot of trouble. You can even use something like Buildroot.

As far as loading it from the network easily, I would also look into Unified Kernel Images (UKIs). They combine the EFI stub, the kernel, and the initramfs into one single file. You should be able to load that directly from the PXE firmware.

[hardwaresofton]

Yeah I'm famliar with initramfs-es (well at least enough to be dangerous)! I've run Alpine from ram before but it just wasn't stable and I ended up going back to a more traditional and perfectly fine setup.

> If you want to run the OS from RAM, the absolute simplest way to do it is to simply never transfer. Building a custom initramfs has never been easier. There are tools explicitly geared towards making an initramfs, like Dracut, but also a huge ecosystem of tools for building container images which could almost certainly be scripted to spit out a cpio without a lot of trouble. You can even use something like Buildroot.

This certainly seems like a lot of responsibility to take on, I'm just surprised there isn't already a distro that is very good at doing everything it does (with modern ease of use) with the caveat of the OS disk being in RAM. Technically there are (there's a whole list on Wikipedia), but last I tried it just didn't work well for me.

These days there are more build-your-own distro options like CoreOS, Flatcar, linuxkit that absolutely make things even easier but they don't have that final bit of being runnable from RAM.

I question whether it's worth it to maintain my own UEFI/initramfs/boot setup rather than just building (or pulling off the shelf) an immutable OS image and using the reset + wipe automation to flash the disks once in a blue-moon when reconfiguring/scaling machines up/down.

> As far as loading it from the network easily, I would also look into Unified Kernel Images (UKIs). They combine the EFI stub, the kernel, and the initramfs into one single file. You should be able to load that directly from the PXE firmware.

This is certainly interesting -- haven't looked into these much, but this would certainly be useful in the PXE firmware (or custom UEFI). But my problem here is that iPXE is dependent on support at the provider level (and Hetzner does not make this available to end users, though they use it internally obviously when you reset), so the other commenter's suggestion:

> replace the PXE stack with an OS installer written in UEFI.

Would not work with this approach. The suggestion is interesting though, custom installer that pulls down a UKI sounds viable from my particular armchair.

[wazdra]

Tails works like this, and has gained a lot of traction in the past few years — although one may argue that running from RAM is only indirectly responsible for its popularity.

I think this idea appeals to many people, also concerning remanence: keeping your system and user data separate to the point that you could virtually mount your /home on any given UNIX host, with the added bonus that if the host is not compatible with your setup, you can always reboot it on your USB stick, run a live ISO on RAM, and retrieve a decent work environment.

[dfe]

True PXE doesn't require coordination with the DHCP server.

https://en.wikipedia.org/wiki/Preboot_Execution_Environment

The network client boot stack sends a DHCPDISCOVER as a broadcast. Any machine can be listening on UDP 67 (bootps) for this. The real DHCP server responds with the DHCPOFFER containing the IP address the client should use. Around the same time, the PXE server responds with its own DHCPOFFER that does not issue an address, but does contain the values for the requested DHCP options.

The client basically keeps broadcasting DHCPDISCOVER until it gets both, then it does the unicast DHCPREQUEST and wait for unicast DHCPACK with the normal DHCP server.

Now, that said, I've only ever seen this work with commercial PXE servers like Microsoft RIS. To my knowledge, ISC DHCPD is unable to send a DHCPOFFER with options but no address. But my knowledge is at least a decade out of date.

At home I just set the options on the main DHCP server like every other hobbyist does, but this isn't true PXE, this is just plain old DHCP+TFTP remote boot.

Let's say you do have such a server that sends DHCPOFFER with the options and no IP address. If it's on its own machine, then it can listen on port 67, same as the real DHCP server on another machine. But, if it's on the same machine as the DHCP server, it has to listen on port 4011. In this case the client behaves a little differently. For this to work, the DHCP server must send as part of the DHCPOFFER an unsolicited option 60 to indicate that the client should go ahead and accept the IP then send a second unicast DHCPDISCOVER to port 4011 and await a DHCPOFFER from that port. Option 60 is only needed, and can only be used, if the independent PXE server is running on the same host as the DHCP server.

So there's basically 3 scenarios: * Hobbyist: just configure the booting options on the real DHCP server * Real PXE, separate machine: Both real DHCP and PXE listen for broadcast DHCPDISCOVER and respond with complementary DHCPOFFER. Real DHCP server has no knowledge whatsoever about booting. * Real PXE, same machine: Real DHCP server responds with unsolicited option 60 no matter what. This is the extend of its knowledge of booting. Separate PXE server runs on port 4011 instead of 67, and everything is unicast.

There may finally be hobbyist projects that support this model, but when I last did this stuff, there were not. Learning how RIS worked was a revelation for me, and it really made me wonder why the hobbyist community of the time seemed hellbent on not doing PXE correctly, which annoyingly requires control over the options set by the real DHCP server, and often makes it impossible to do fun stuff like use different boot files for different clients.