[latchkey]

For me, the pool of people to hire that know Linux inside and out would be much larger. This is worth any perceived security issues.

In terms of diskless, I've run 25k+ iPXE deployments on diskless blade servers using a highly customized Ubuntu, and it was fantastic.

Regardless of OS choice, being diskless is also quite nice... if there was a security issue or you need an upgrade of some sort, you just reboot. Only thing is that it takes a while to reboot 25k servers... even on gigE. It was a bit of work to build the scheduling system to make that happen reliably, but it worked out quite well.

[ckdarby]

I guess that is some of their focus around why they got their image down to 200MB.

Even better if you had boxes with 10 gigE and the smaller image. Would take your times down from like 6-10 hours to 1.5 hours.

Also, I doubt a full 25k restart all at once you probably had underlying applications that expected rolling, blue/green or even % or nodes that can go offline at once.

[latchkey]

Full boot was around 160MB and that included AMD drivers, which were about 60MB.

I did not have a choice over the hardware design.

Each worker was individual. Underlying application didn’t care.

We had a couple full power outages. They were set to boot automatically. So yea… full restarts.

[infofarmer]

Not sure the actual authors of the various overlapping Linux network subsystems even know the comprehensive picture "inside and out" for chronic lack of consistent documentation.

Last time I managed a small «supercomputer», 50x IBM blades running Suse, it wouldn't support PXE/NFS without kernel customization, but that would void support contracts and finicky third-party software. Made a switch to FreeBSD, where everything worked out of the box one hour later. That was over 15 years ago, I have no idea how much the situation changed.

[latchkey]

Things have improved and ubuntu is better than suse. =)

This was effectively 25k PS5's... much more powerful now.