[encabulator]

I just finished another late night "thanks" to unknown person who decided to use this approach several years ago while setting up 17 servers that I inherited. The /boot partition of these RHEL machines was put on SD cards, which have been gradually dying. So I got to boot the machine (which is 6000 km away) to recovery mode from ISO and re-create the /boot filesystem on HDD (RAID).

Of the 17 servers, only three are now remaining where I haven't had to do this. Not all of these were actual SD card failures, some where done preventively. Still, there have been several SD card failures requiring emergency repair work at inconvenient times. There have been 0 RAID failures requiring similar emergency work on the systems where /boot has been migrated to HDD based RAID.

Of course there is no replacement SD card handy with working /boot filesystem. Actually there is no such thing as "handy" in my case. I can actually reconstruct /boot partition on the HDD faster than anybody could go to the datacenter and replace the SD card. And if there were a replacement SD card, I would need to keep it up to date manually every time the kernel or initrd are updated.

I never want to see this kind of setup again, and frankly it felt insane the first time I saw it.

Of course I should have migrated all of the machines off the SD cards by now, but my excuse is that maintaining Linux on these machines is not really my responsibility (it is nobody's responsibility apparently, although many people are interested in keeping these systems online).

When I die I want to have a SD-card shaped tombstone.

[StreamBright]

So you discovered the well known problem in "IoT" that SD cards should never be used for filesystems that are online 100% of the time?

I think the parent views on storage is still valid. Do not boot from the RAID volume where you store many terabytes.

I usually have a small mirror SSD for the OS (like 128G or max 256G). Those drives are super cheap, easy to replace.

For the /data I usually use ZFS or mdadm + xfs. Depending on the use case (for example do you want to add more capacity later? ) you can decide.

Using SDcards for server filesystem is negligent at the very least.

[krageon]

flash dying after a few years is not exactly weird or unexpected, regardless of the form factor

[mattpallissard]

This was a trend for a minute back in the late 00's. Any system that was

  1.HA
  2.Had an OS that would get fully loaded into memory (think ESX).

The idea was that SD-cards take less power and where cheap enough you could have a whole gang of spares with images ready to just drop in place. I'm not saying it was a good idea, just that I've encountered this more than once.