[TickleSteve]

Thats a well known issue with standard distributions, and there are fairly well known ways around it...

Basically, don't do continuous writing to the card, turn off atime updates,possibly even work entirely from the initrd if possible. Also, use proper industrial sd cards, these have proper wear-levelling and bad-block management unlike the consumer cards.

In other words, treat it as an embedded system and tailor it to your use-case.

[sgt]

I've tried all these "ways around it", but SD card corruption still happens on Raspberry Pi's. We use them as dashboards around the office so having them running 24/7 really tests these devices.

I've found that the only working recipe for a stable Raspberry Pi is a USB hard drive (prefer SSD but mechanical also fine), and a UPS.

My simple advice is: do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale, as you will run into filesystem corruption and create a lot of work for yourself.

[dividuum]

> do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale

Can't confirm. I've written about this before, but I'm mainly responsible for running the https://info-beamer.com digital signage service. It's using a custom Linux distribution that always boots into an R/O system. All user content is on a separate partition that can always be restored if there's any error. The ext4 fs is tuned in a way that the kernel defers writes quite a while to minimize write access if possible. Since customers purchase their own SD cards, we have limited control of what ends up in a device. I've seen exactly one SD card related problem thus far. There are devices that have been running >3 years 24/7, so it's doable. But you can't just use Raspbian. You have to properly design the system.

[sgt]

Thanks - that does give me hope.

[TickleSteve]

I agree, which is why in a properly designed system you would run from initrd, its a single purpose system after all. No need for any sd card access once the image has been read into RAM. The SD card is read-only in this case. Also, I wouldnt trust consumer sd cards at all, they typically lack to necessary features to store data reliably anyway.

[bonzini]

Or just make your root file system read-only. Running off an initrd greatly limits your filesystem size.

[sgt]

Would TinyCore be a good option for us? We just need to run RPi's and Chromium to display a web page.

[elcritch]

Not familiar with TinyCore myself, but I’d recommend trying Nerves Kiosk if a webpage kiosk is your primary goal [1]. Getting all of the FS settings right and a tuned system like daviduum talks about is tricky IMHO. Though it should be possible with Yocto Linux or similar as well I’d imagine, especially with ‘fwup’ or similar in the loop. Nerves uses ‘fwup’ [2] to create a R/O squashfs filesystem for the core OS image which is also highly compressed.

1: https://github.com/LeToteTeam/kiosk_system_rpi3/blob/master/... 2: https://github.com/fhunleth/fwup

[rsync]

"My simple advice is: do not try to run Raspberry Pi from SD cards in "production" under any circumstances on a large scale, as you will run into filesystem corruption and create a lot of work for yourself."

Is that true even if you mount the filesystem(s) read-only after boot time ?

As an aside, I have to say that it is very surprising and annoying how fragile "modern" drive parts are compared to decades ago when we were using either disk-on-chip or just plain old CF cards pin-compatible with the IDE connector.

I had 16MB CF cards on the IDE bus that lasted 15+ years[1] in production without a hiccup, but in 2018 SATA SSDs die randomly from time to time ...

[1] Yes, they were all mounted read-only ...

[mrsteveman1]

> Is that true even if you mount the filesystem(s) read-only after boot time ?

Yes, read-only filesystems are not the solution to most of the Micro SD card problems on the Pi, because the filesystem and the OS are relying on the card to work properly, and the actual consumer market Micro SD cards people are using don't always do that when used in the Pi.

To manage the flash, the (dirt cheap, barely fit for purpose) controller inside a consumer Micro SD card will generally do things like move data from one area of flash to another, even when the host isn't writing to the card.

One of the reasons for that, is that reading from NAND is a potentially disruptive[1] event:

> If reading continually from one cell, that cell will not fail but rather one of the surrounding cells on a subsequent read. To avoid the read disturb problem the flash controller will typically count the total number of reads to a block since the last erase. When the count exceeds a target limit, the affected block is copied over to a new block, erased, then released to the block pool.

There are industrial Micro SD cards that can handle most of the problems that show up on the Pi though, and they aren't absurdly expensive. The $15 ATP AF4-GUD3A and $28 ATP AF8-GUD3A are some I use in my Pi projects for this reason, and I've never seen any of them become corrupt, unlike SanDisk and other consumer cards.

[1] https://en.wikipedia.org/wiki/Flash_memory#Read_disturb

[ac29]

> As an aside, I have to say that it is very surprising and annoying how fragile "modern" drive parts are compared to decades ago

Those old parts used SLC (single bit per cell) flash, which has orders of magnitude higher endurance than whats used now, MLC or TLC (2 and 3 bits per cell). QLC (4 bit per cell) is starting to hit the market now.

[gsich]

You can network boot them too, which is way more convinient. RPi3 can do it natively, the former ones need an SD card, but since it's only for the bootloader, it'll survive much longer.

[dividuum]

Last time I read about this [1], the firmware had some issues that prevented the boot process from working 100% reliable. Do you have any experience with network booting the RPi?

[1] https://github.com/raspberrypi/firmware/issues/764

[gsich]

Yes, but I only have 2 devices that network boot. One is with PoE, one without. Both work fine, but I don't have the need to reboot them every other day and the network is not a complex structure. Filesystem is with NFS.

[limaoscarjuliet]

I use raspberry pi as a display around a mid sized company. Out of 12 setups, I have to resurrect 2-3 every year due to ssd corruption.

[sgt]

Are these using UPS or just directly on power? At our office, power goes off about once or twice a month when someone is working on the electrical system. We've had lots of building work done, and it's definitely not been good on the Pi's - that being until we added UPS's.

[kingosticks]

Can I ask what industrial sd cards did you try? I was going to try one of those out but I'd like to know what to avoid. Thanks.

[xeeeeeeeeeeenu]

If you want reliability, buy SLC SD cards. 'SLC' keyword is much more important than the brand. There are a few companies that make them, for example Panasonic or Swissbit.

[nickpsecurity]

Good point. To elaborate, SLC means Single-Level, Cell, NAND Flash that's both simpler and longer-lasting:

https://en.wikipedia.org/wiki/Multi-level_cell#Single-level_...

Example suppliers:

https://www.micron.com/products/nand-flash/slc-nand

https://www.cactus-tech.com/resources/blog/details/why-slc-s...

http://www.macronix.com/en-us/products/NAND-Flash/SLC-NAND-F...

https://business.toshiba-memory.com/en-us/product/memory/slc...

[sgt]

We primarily used class 10 consumer cards including Kingston, Sandisk, Samsung etc. We also tried ATP which is sold as an industrial SD card. More info: https://www.atpinc.com/products/industrial-sd-cards

[jgalt212]

are there ssd or winchester drives that can plug into the sd card slot? Of course, the drive would need an additional power supply.

Or maybe this is just over-engineered, and maybe someone makes a pi that uses an SSD by default.

[Koshkin]

Not sure about the SD card slot, but the recent Pi’s can boot from a hard drive attached to a USB port.

[avip]

Not to downplay the idea that pi's fs in unreliable, but there's another player here which is the SD card. Many SD cards are way less reliable than the imaginary R/W spec given by manufacturers, and there's a crazy amount of grey matter and counterfeiting.

[anfractuosity]

atime refers to timestamping of files with their access time right?

That sounds interesting re. turning off atime updates, how do you do that out of interest?

[Fnoord]

> how do you do that out of interest?

Mount with -o noatime. The default (-o relatime) should however be good enough for most situations.

[rad_gruchalski]

noatime mount option