[chocobor]

Q: How do you guys centrally update field devices?

I am working on professionalizing our IOT setup that currently consists of a few dozen raspberries which run docker containers. They are individually updated by sshing into them and running apt update manually. Docker containers are deployed with a commercial solution. I want to have a centralized way to update the OSes, but it does not really make sense for our small team to introduce yocto knowledge, because that would make us fall behind development schedule even more. Also, the hardware needs are just too boring to justify rolling our own os. I have not yet found a hardware independent Linux distro that can be reliably updated in an IOT context.

I am now looking if we can buy ourselves out of this problem. Ubuntu Core goes in the right direction, but we don't want to make us dependent on the snap store. Advantech has a solution for central device management with Ota updates, maybe we are going that route.

How do you guys update field devices centrally? Thanks!

[PhilipRoman]

Not what you're looking for, but https://sbabic.github.io/swupdate/swupdate.html

It's meant (I think?) for immutable style distros like Yocto. You basically create a cpio archive and a manifest of what file goes in which partition (plus bells and whistles like cryptography). It's a good idea to have double buffering, so that if boot fails to come to a reasonable state, the device will revert after a few tries.

IMO the mutable distro model is way too fragile for long term automated updated. Errors and irregularities accumulate with each change. Besides, the whole "update while the system is running" is actually not well defined behaviour even for Linux, it just happens to work most of the time.

[cushychicken]

+1 for swupdate. Implemented it three or so different times at this point and really like it.

[unitexe]

I have also implemented it 3 times. Local updates from file system or USB and networked updates with the mongoose web page & the REST API. Quirky to configure when supporting all these methods at the same time but smooth and reliable once done.

Never got to implement a full OTA update system. SWUpdate supports suricatta & Siemens WFX for this. I played around with WFX and got OS updates working on my local network but didn't do anything else with it.

Nonetheless, if I wanted to implement my own OTA update solution swupdate & WFX seem like a beautiful combination.

If I may ask, how far did you take it?

[cushychicken]

I’ve only ever done hard connect updates over USB and manually pushed updates over local network. The team I was on was on track to switch over to pulled updates using Suricatta but I left before they switched them on.

[unitexe]

Cool, thanks for sharing!

[leonheld]

> have not yet found a hardware independent Linux distro that can be reliably updated in an IOT context

I'm part of the team that builds an immutable distro based on OSTree (https://www.torizon.io) that does exactly that.

Docker/Podman support is first-class as the distro is just a binary, Yocto-based one that we maintain so users don't have to. You can try our cloud for free with the "maker" tier. To update a device you just drop a compose file with the web ui and massively update a fleet. You can even use hardware acceleration from the containers using our reference OCI images.

The layer is open (https://github.com/torizon/meta-toradex-torizon) and will get Raspberry Pi support soon but you can integrate already easily with meta-raspberrypi (we can also do this for you very quickly ;-)).

Happy to answer any questions.

[mort96]

I've used RAUC (https://rauc.io/) professionally for a couple of projects and am happy with it. There's a RAUC meta layer which provides bbclasses for generating rauc bundles from an image recipe. It's not that complicated to set up boot partition selection in u-boot.

For embedded systems, I strongly prefer the "full immutable system image update" approach over the "update individual packages with a package manager" approach. Plus you get rollbacks "for free": if the system doesn't boot into the new image, it automatically falls back to booting into the previous image.

[mvip]

+1 for "full immutable system image update"

People who suggest updating individual packages (or even worse, individual deb packages for instance) have never deployed any large scale IoT/Embedded projects. These devices are very different than servers/desktops and will break in ways you can't imagine. We started out using deb packages at Screenly before moving to Ubuntu Core, and the amount of error/recovery logic we had written to recover from broken deb package state was insane at that point.

[unitexe]

One has to implement the rollbacks in the bootloader after they have implemented an A/B update pattern.

In u-boot this is done with its boot count limit config and altbootcmd.

[nullify88]

I would suggest taking a look at bootc https://github.com/containers/bootc which enables you to use OCI / Docker containers as a transport and delivery system for OS updates. That makes available much of the tooling used to build and deliver container images for the purposes of delivering os updates.

Such possibilities include the various registeries available for storing OS updates and branches. Tooling for security scanning, sbom generation, signing Docker or podman for building the image.

It's important to note that the container image itself is not executed upon boot, but rather unpacked before hand.

[mvip]

I would look at Balena if you are already using Raspberry Pi’s and docker. Alternatively maybe look into ROAC but don’t know if it supports docker. The SD cards will be your biggest failure point, so select them wisely.

I’ve deployed Ubuntu Core at scale. It’s great but does have its learning curve. Theirs is also somewhat of a lock in, even if you can run everting yourself. However, their security is really good.

[thebruce87m]

Yocto + Mender is one option, but you don’t the yocto pain. We are trying Balena at the moment and liking it. It manages both the OS and the docker bit.

[palata]

With Balena you are shipping entire Linux distros that you did not build, right? How do you deal with licences?

E.g. if you ship an Ubuntu container, you have to honour the licences of all the packages that you are shipping inside that Ubuntu container. Do you?