[OnACoffeeBreak]

I would guess pacemakers don't run Linux. I would be surprised if they run an OS at all. I work on devices that have to survive 20 years on one non-rechargeable and non-serviceable battery, and there's at most a simple scheduler in place to control tasks. We use 32 bits for epoch time, but our epoch starts Jan 1, 2000, so we have 30 years on Linux before this becomes a problem.

[zeptomu]

This is really interesting! Obviously it is a topic in itself, but do you mind sketching how one develops software in an environment that requires this extreme amount of reliability? If you do not use an existing general-purpose OS, do you even use a MMU? Is it hard real-time?

E.g. what language do you use? Is it SW or HW that you "ship"? You probably perform some kind of verification and or validation - how does the tool chain look like?

Do you perform model-checking on all possible inputs?

Lots of questions, and you do not have to go into detail, but I would appreciate your input, as it is an interesting topic.

[OnACoffeeBreak]

No MMU. It is hard real-time in the sense that there are events that need to be processed withing a small time window (a few microseconds (with help from hardware typically) to milliseconds).

The product is custom hardware built with off-the-shelf parts like microcontroller, power converters, sensors, memory. Texas Instruments MSP430 family of microcontrollers [1] is popular for this type of application. They are based around MIPS CPU cores with a bunch of peripherals like analog-to-digital converters, timers, counters, flash, RAM, etc.

I don't work on medical devices, so validation is more inline with normal product validation. We certainly have several very well staffed test teams: one for product-level firmware, one for end-to-end solution verification, others for other pieces of the overall solution. We are also heavy on testing reliability over environmental conditions: temperature, pressure, moisture, soil composition, etc.

The firmware is all done in-house written in C. Once in a while someone looks at what the assembler the compiler, but nobody writes assembler to gain efficiency. We rely on microcontroller vendor's libraries for low-level hardware abstraction (HAL), but other than that the code is ours. The tool chain is based on GCC I believe, but the microcontroller vendor configures everything so that it crosscompiles to the target platform on PC.

Debugging is done by attaching to the target microcontroller through a JTAG interface and stepping through code, dumping memory, checking register settings. We also use serial interfaces, but the latency introduced by dumping data to the serial port can be too much for the problem we're trying to debug and we have to use things like togging IO pins on the micro.

We don't model the hardware and firmware and don't do exhaustive all possible inputs test like one would do in FPGA or ASIC verification.

I need to go, but if you have more questions, feel free to ask, and I'll reply in a few hours.

1: http://www.ti.com/lsds/ti/microcontrollers-16-bit-32-bit/msp...

[zeptomu]

Thank you for your thorough answer.

I am surprised that you do not apply some kind of verification or checking using formal methods, however it might be the case (at least it is the experience I have) that this is still too inconvenient (and so expensive) to do for more complex pieces of software.

[nickpsecurity]

Actually, the high-assurance field that does such things is very small. A tiny niche of the overall industry. Most people doing embedded systems do things like the parent described. The few doing formal usually are trying to achieve a certification that wants to see (a) specific activities performed or (b) no errors. Failures mean expensive recertifications. Examples include EAL5+, esp DO-178B or DO-178C, SIL, and so on. Industries include aerospace, railways, defense, automotive, supposedly medical but I don't remember a specific one. CompSci people try formal methods on both toy, industrial, and FOSS designs all the time with some of their work benefiting stuff in the field. There's barely any uptake despite proven benefits, though. :(

For your pleasure, I did dig up a case study on using formal methods on a pacemaker since I think someone mentioned it upthread.

http://www.comp.nus.edu.sg/~pat/publications/ssiri10_pacemak...

David Wheeler has the best page on tools available:

https://www.dwheeler.com/essays/high-assurance-floss.html

Here's a work-in-progress of my list of all categories of methods for improving correctness from high-assurance security that were also field-proven:

https://lobste.rs/s/mhqf7p/static_typing_will_not_save_us_fr...

[OnACoffeeBreak]

One important thing to note is that the 20-year life expectancy includes several firmware updates. An update may take several hours to several days to complete, so, it's not something that is commonly done, but it's an option.

I am fairly new to this field and I share your surprise that more formal methods are not used in development. To be honest, the development process in my group and others I'm familiar with can be improved tremendously with just good software development practices like code reviews and improved debugging tools.

[pinaceae]

you might be underestimating modern med devices which even have wireless access (beats a port or invasive surgery to read a log).

example for the issues in this area: https://spqr.eecs.umich.edu/papers/49SS2-3_burleson.pdf

[OnACoffeeBreak]

I see "wireless", but not "WiFi" or "802.11*" in the PDF.

For what it's worth, devices I work on have a few wireless interfaces while guaranteeing 20-year life time: one interface is long-range (on the order of 10km), two are short range (on the order of a few mm). There is no way we can get to 20-year life time with doing WiFi (maintaining current battery size/capacity) for long'ish range and maybe not even BT for shorter range.

[pinaceae]

yes - but this means there is more than a simple OS operating in those devices.

and just like any other IoT, using generic chips and stacks is cheaper.

run QNX or Linux on it and walk away.

there are DYI insulin pump monitors out there already that use Linux on RaspberryPi - see here: https://openaps.org/

[OnACoffeeBreak]

The microcontrollers on these devices don't have MMUs. There is typically not even a USB interface. The microcontroller is in deep sleep mode saving power 99.9% of the time. During that time only essential peripherals are powered on and no code is executing.

A RasPi has no chance of running for 20 years off a single A-size non-rechargeable non-serviceable battery.