I've had numerous failures on older RPi models due to SD card corruption, even in non-write intensive contexts. Not sure this is good enough when you want reliability...
Thats a well known issue with standard distributions, and there are fairly well known ways around it...
Basically, don't do continuous writing to the card, turn off atime updates,possibly even work entirely from the initrd if possible.
Also, use proper industrial sd cards, these have proper wear-levelling and bad-block management unlike the consumer cards.
In other words, treat it as an embedded system and tailor it to your use-case.
I've tried all these "ways around it", but SD card corruption still happens on Raspberry Pi's. We use them as dashboards around the office so having them running 24/7 really tests these devices.
I've found that the only working recipe for a stable Raspberry Pi is a USB hard drive (prefer SSD but mechanical also fine), and a UPS.
My simple advice is: do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale, as you will run into filesystem corruption and create a lot of work for yourself.
> do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale
Can't confirm. I've written about this before, but I'm mainly responsible for running the https://info-beamer.com digital signage service. It's using a custom Linux distribution that always boots into an R/O system. All user content is on a separate partition that can always be restored if there's any error. The ext4 fs is tuned in a way that the kernel defers writes quite a while to minimize write access if possible. Since customers purchase their own SD cards, we have limited control of what ends up in a device. I've seen exactly one SD card related problem thus far. There are devices that have been running >3 years 24/7, so it's doable. But you can't just use Raspbian. You have to properly design the system.
I agree, which is why in a properly designed system you would run from initrd, its a single purpose system after all.
No need for any sd card access once the image has been read into RAM. The SD card is read-only in this case.
Also, I wouldnt trust consumer sd cards at all, they typically lack to necessary features to store data reliably anyway.
Not familiar with TinyCore myself, but I’d recommend trying Nerves Kiosk if a webpage kiosk is your primary goal [1]. Getting all of the FS settings right and a tuned system like daviduum talks about is tricky IMHO. Though it should be possible with Yocto Linux or similar as well I’d imagine, especially with ‘fwup’ or similar in the loop. Nerves uses ‘fwup’ [2] to create a R/O squashfs filesystem for the core OS image which is also highly compressed.
"My simple advice is: do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale, as you will run into filesystem corruption and create a lot of work for yourself."
Is that true even if you mount the filesystem(s) read-only after boot time ?
As an aside, I have to say that it is very surprising and annoying how fragile "modern" drive parts are compared to decades ago when we were using either disk-on-chip or just plain old CF cards pin-compatible with the IDE connector.
I had 16MB CF cards on the IDE bus that lasted 15+ years[1] in production without a hiccup, but in 2018 SATA SSDs die randomly from time to time ...
> Is that true even if you mount the filesystem(s) read-only after boot time ?
Yes, read-only filesystems are not the solution to most of the Micro SD card problems on the Pi, because the filesystem and the OS are relying on the card to work properly, and the actual consumer market Micro SD cards people are using don't always do that when used in the Pi.
To manage the flash, the (dirt cheap, barely fit for purpose) controller inside a consumer Micro SD card will generally do things like move data from one area of flash to another, even when the host isn't writing to the card.
One of the reasons for that, is that reading from NAND is a potentially disruptive[1] event:
> If reading continually from one cell, that cell will not fail but rather one of the surrounding cells on a subsequent read. To avoid the read disturb problem the flash controller will typically count the total number of reads to a block since the last erase. When the count exceeds a target limit, the affected block is copied over to a new block, erased, then released to the block pool.
There are industrial Micro SD cards that can handle most of the problems that show up on the Pi though, and they aren't absurdly expensive. The $15 ATP AF4-GUD3A and $28 ATP AF8-GUD3A are some I use in my Pi projects for this reason, and I've never seen any of them become corrupt, unlike SanDisk and other consumer cards.
> As an aside, I have to say that it is very surprising and annoying how fragile "modern" drive parts are compared to decades ago
Those old parts used SLC (single bit per cell) flash, which has orders of magnitude higher endurance than whats used now, MLC or TLC (2 and 3 bits per cell). QLC (4 bit per cell) is starting to hit the market now.
You can network boot them too, which is way more convinient. RPi3 can do it natively, the former ones need an SD card, but since it's only for the bootloader, it'll survive much longer.
Last time I read about this [1], the firmware had some issues that prevented the boot process from working 100% reliable. Do you have any experience with network booting the RPi?
Yes, but I only have 2 devices that network boot. One is with PoE, one without. Both work fine, but I don't have the need to reboot them every other day and the network is not a complex structure. Filesystem is with NFS.
Are these using UPS or just directly on power? At our office, power goes off about once or twice a month when someone is working on the electrical system. We've had lots of building work done, and it's definitely not been good on the Pi's - that being until we added UPS's.
If you want reliability, buy SLC SD cards. 'SLC' keyword is much more important than the brand. There are a few companies that make them, for example Panasonic or Swissbit.
We primarily used class 10 consumer cards including Kingston, Sandisk, Samsung etc. We also tried ATP which is sold as an industrial SD card. More info: https://www.atpinc.com/products/industrial-sd-cards
Not to downplay the idea that pi's fs in unreliable, but there's another player here which is the SD card. Many SD cards are way less reliable than the imaginary R/W spec given by manufacturers, and there's a crazy amount of grey matter and counterfeiting.
I'm pretty sure my Pi failures where due to weak power supplies. I tried running a Raspberry Pi 1 B on a 1 A power supply, and after a few months it wouldn't boot. Since then I've switched to a bigger SD card (16 GB) for better wear leveling, and a 2,5 A power supply and haven't had any issues yet.
It's often a combination of the power supply and the type of SD cards, but eventually you will have a failure no matter what SD card you use. I've learned this the hard way from running dozens of Pi's 24/7. Looking back, I regret the decision to use Raspberry Pi. See my comment below.
Basically, don't do continuous writing to the card, turn off atime updates,possibly even work entirely from the initrd if possible. Also, use proper industrial sd cards, these have proper wear-levelling and bad-block management unlike the consumer cards.
In other words, treat it as an embedded system and tailor it to your use-case.