[NAHWheatCracker]

I have some Raspberry Pis that I self host stuff on. Gitlab on one, Runtipi on another, and pi-hole + Nginx + a game I made on a third. Some of it is publicly accessible on my own domains, but the only traffic I actually get is from myself.

I connect my Pis to my WiFi router and I've struggled with the adapters, so a POE switch might be interesting. I should containerize stuff, but I enjoy running it directly as well. I need to figure out backups some day.

The biggest issue I've had is that I've lost 3 SD cards in 6 years. Twice due to power failure, and once to heavy disk writes. I can't stand randomly losing stuff and having to re-setup a pi, so I switched to booting from USB SSDs. I haven't had an issue since. The cost is a bit obnoxious since I don't trust a cheap drives, but the perceived peace of mind is nice.

Has anyone else had this issues with losing SD cards while self hosting on Pis?

[bnabholz]

As to your direct question: I used some Pis for TV dashboards at work and had some random bit flips on the SD card and corrupted files for the dashboards. It may be "rare," but seems inevitable on a long enough timeline. For toy projects where you can re-image the SD card it's alright, but even for my trivial personal projects it made me uneasy.

100% to backups. I know we all put off doing it, but you'll rest a lot easier, even with personal data you don't think you care about. It's not only about a hardware failure, but even a fluke sysadmin error where you accidentally nuke something. I'd recommend getting a account for Backblaze B2, and setting up restic on each Pi to at least daily backup the data directories and stuff you care about. For your Gitlab it's a bit less risky since presumably you also have a clone of each repo on some other machine.

I love that people are building small datacenters out of Pis. I haven't done the math as far as TCO, but instead of multiple Pis for self-hosting, I have a lonely secondhand Dell Precision with an old 8th gen Intel CPU (6C/12T), 64GB of RAM, and several TB of NVMe plus some spinning rust for the long term stuff. It's just a crazy amount of horsepower. Most trusted workloads run as containers, and my other experiments can run as VMs, and I have capacity in all the right places (I need disk and RAM more than CPU). Not as exciting as building a cluster, but I have the excess capacity to spin up multiple VMs on that one machine, if I want to play with that. It can get very Inception-like, what if I'm running VMs in KubeVirt on top of Kubernetes that is running on a cluster of VMs that are ultimately on a single machine, but while delegating whatever extra /64 IPv6 prefixes Comcast gave me to each of the bottom-layer VMs so that each pod still gets a globally routable IPv6 address. Cool times for the homelab stuff, and helped me understand things like Kubernetes and IPv6 to a much greater depth.

[vanilla_nut]

Only time I've ever had an issue is when I manually disconnected power from a Pi 4 when I overreacted to a network outage. I was dumb and thought my Pi's DNS was screwing things up for the whole network, but it turns out that it was just Spectrum screwing things up for the entire region. Fortunately there was only one minor corrupted file, it didn't bork the SD card, and I was able to repair the setup manually.

I've never had any problem as long as I've stuck with the `sudo shutdown` command, which powers down in a controlled manner so writes don't get interrupted. But I've also never had an issue after a power outage, so I think somehow power outage shutdowns are more graceful than just yanking the power cable out.

If you're really concerned, you can always configure your Pi to use a read-only root filesystem. Combine that with a USB SSD to store your _actual_ data and you should be OK indefinitely. Or just boot from that USB SSD -- just remember to configure fstab correctly for your intended behaviour! You might be surprised to discover that fstab can delay a boot indefinitely if mounting expectations do not match reality.

[jeromechoo]

I’ve had 3 SD cards fail on me in the last year. I now avoid using them as serious long term storage.

To be fair, these SD cards were exposed to fairly extreme Texas temperatures. One in a car dashcam, the other in an outdoor camera.