[rwmj]

> If the backup camera or other sensors were to run as containers, we needed to improve the starting speed significantly.

I think I see the problem already. Why does anyone think its a good idea to put everything in an embedded system into a container? Particularly as everything comes from a single vendor and so the usual argument about "but libraries are too hard!" doesn't apply.

[erinaceousjones]

To be honest I think we should be adopting the full ecosystem we've been busy building around containerization.

Imagine calling up breakdown assistance because your car won't start, mechanic comes out, cracks the hood and is like "ah there's your problem right there, ignition service has only 1/2 pods healthy because the node went into NotReady due to DiskPressure. I can clear up some log files so it goes underneath 80% disk usage again but sucks teeth it's gonna cost ya. I'd recommend throwing the whole car out and getting a new one. You shouldn't get an attachment to these things, they're cattle not pets."

Truly breathtaking.

[fredsted]

Looks like your cars Kubernetes certificates have expired after a year, we'll need to SSH in and run kubeadm to refresh them. Wait, the 5G pod isn't starting ...

[wpietri]

You joke, but I decided I wanted to try out Kubernetes, so I set it up at home and moved some of my local services into it, one of them being my custom lighting automation software. Eventually my lights just stopped working, and after a lot of rummaging it turned out to be expired certificates. I promptly put the software back where it was before, removed Kubernetes, and decided there were better things for me to play around with.

[yellowapple]

Can't wait to plug in an ODB2 scanner and get a k9s screen.

[andix]

I agree. Microservices are a great concept for a lot of problems. But now I constantly see way too small services. Every tiny piece of software gets its own service and its own software lifecycle (including versioning and deployment).

And what happens now is, that you need a huge effort to integrate all those components. End to end system tests get much more important, but are still harder to do than simple unit/integration tests. traditional testing strategies start to get pointless, because most bugs now only appear when combining services in a production setup.

Yes, development gets easier, because every team can just develop, without aligning too much with other teams. But the deployment/ops/acceptance step often gets impossibly complex.

[zokier]

You are blaming cultural issues on technology. You have the same integration pains with libraries too. In general if your teams are not aligned, it doesn't really matter if they are developing microservices, libraries, or one spaghetti ball mess, its going to be problem anyways

[ilyt]

The old method of forcing everyone to freeze at certain dependencies also means that any bug in those dependencies is fixed in all components at once.

Obviously going too big here is problematic as it can slow it down when tens or hundreds of people are involved in every update, but going too small have similar problems, on top of generally more smaller services eating more resources. We don't need "front reflector LED setting app" being called from "lighting setting app" called from "car setting app", it can probably just be one service.

Smaller services also mean more services to update if some commonly used lib gets a security bug. Updating SSL lib in big monolith is just update, run tests, but in microservices that's multiplied by amount of teams and services.

> In general if your teams are not aligned, it doesn't really matter if they are developing microservices, libraries, or one spaghetti ball mess, its going to be problem anyways

Moot point. We pick the tools to make the job easier. Good team with bad tools will still be slower and less efficient than good team with good tools.

[goodpoint]

> You are blaming cultural issues on technology

Because technology almost always carries cultural values with it.

[intelVISA]

Excessive microservices is lazy engineering. It doesn't solve any problems - they just get moved to the next quarter, or some other team.

[vbezhenar]

Do you oppose running camera software in a separate process? I think it makes sense. Camera process might crash and will be restarted, this should not cause restart of the entire shell.

What people should understand is that container in Linux is just a separate process running in powerful chroot (which isolates not just file tree, but also process tree and other things).

So the same reasoning which applies to running some code in a separate process also applies to running some code in a separate container.

I'd even argue that in an ideal world, almost every process should run in a separate container. The tooling is not here, but concept is just fine.

[garaetjjte]

Containers usually ship their own libraries, which means less sharing, more disk usage and higher memory pressure.

[vbezhenar]

That depends on implementation. Shared libraries which use the same inode will be shared AFAIK. If containers use different libraries, they'll not be shared, of course, but that's a deliberate choice of container creator.

[bonzini]

Containers can use the same base image for the OS.

[rusk]

I see these as a relatively straightforward set of problems to identify, quantify, and remedy. It’s a tradeoff between static memory usage and stability. If that additional memory footprint becomes an issue you can make plans to align dependencies.

[goodpoint]

And much lower chances of getting security updates, now that everything is a huge blob.

[IshKebab]

It's not just the camera process though - containers ship an entire OS (except the kernel).

[aljarry]

Though you don't need most of the OS - you may run bare bones containers with just a statically built binary inside the image, e.g. it's possible in Go.

[jpgvm]

I think if you open up a Tesla infotainment system you would find something eerily similar to containers. Remember that what you and most programmers think of containers is merely one possible assembly of a bunch of kernel features. There exists not just a gradient between plain processes and containers but a whole solution space with different tradeoffs.

I happen to know a small amount of the Tesla internals and they are using cgroups, namespaces, app armour and ebpf based syscall filtering to secure various processes on the car.

You almost certainly should not use docker or podman to manage processes on a car but that doesn't mean you shouldn't embrace the subsystems they are built on in order to increase security resilience and defense in depth.

[galangalalgol]

Everything never comes from a single vendor. Even microcontrollers the hardware abstraction layer is hiding peripheral support packages for all the i2c and spi things it talks to, each from a different vendor potentially.

I'm not saying we should be running k8 on an 8051 or even a cortex m33, but on an arm7? Maybe.

Cult of Ferris time, static linking in rust means your binary is your container, particularly if you statically link in musl.

[GordonS]

Playing Devil's advocate here, but there are some good reasons you might do this. For example, Docker's "pull" system is great for updates, and means it's trivial to rollback to an earlier version if something went wrong. A Docker registry also means you can easily switch to another version when needed. You also get a supervision of containers, with automatic restarts (yes, I know you can do this nowadays with systemd :)

[andix]

Sure, but if you provide the possibility to deploy a lot of components separately, at some point they will run in different versions. And who knows if rear view camera 1.3.22-44 works with turn signal 4.86.233-stable and break pedal 0.6.9876-beta?

[bluGill]

Those 3 for the most part don't have to work together. They need to run on the same CPU without taking more than their allowed portion of the CPU.

If they have to work together the communication protocol is clearly defined well in advanced and limited to exactly what they need to say. Thus we are reasonably sure if any one combination works all possible combinations will work. Even then there is typically higher level control to only release combinations that are tested to work together.

[andix]

That sounds good in theory. The reality I live in, is different though. If it works for you, that’s great!

[wiseowise]

How is that different from shared libraries?

[solatic]

Shared libraries are bundled into one container, then the one container is submitted to a gauntlet of tests (service / API-level tests, e2e tests, etc.).

If you're submitting an embedded device to a gauntlet of tests, you're anyway "containerizing" all the shared libraries when you build the embedded image. Trying to build containers within the embedded image has questionable merit.

[andix]

It’s the same problem. But library authors usually take extra care to stay compatible.

And if you create your own shared libraries, they are normally not deployed separately, usually you bundle them with your main executable.

[aaravchen]

Because you can prove it. In the auto industry you can't just claim it conforms, you have to prove it does for all possible conditions. With containers enforcing that separation, you can prove it much more easily.

[ClumsyPilot]

A statically compiled executable is even easier, and can be hosted on your webserver

[aaravchen]

But unfortunately they also become very space inefficient when a lot of processes need the same (relatively) large blocks of code. But if you carefully curate your containers to use the same base image that contains those libraries already, you don't have to duplicate it.

Comparable would be to having an inode de-duplicating file systems, and deterministic binary generation. But it's hard to prove "the correctness" of inode de-duplicating file systems in extreme environments like auto is required to, and deterministic binary generation is hard to control 3ven if it is possible with the specific build tools (it usually isn't).

[qprofyeh]

Yup, I can imagine complete images are easier to version-manage and to tag as minor/major/hotfix than each individual part of the stack.

I think people underestimate the amount of software that is ALREADY running in their cars/airplanes/helicopters and even elevators.

[rwmj]

"yum", "apt" etc have registries, roll back etc and have for a good few decades.

[talonx]

<sarcasm>But..but..they are not "cool" </sarcasm>

This forced application of new technologies into every possible domain can have real security and reliability consequences - and all because some VP somewhere decided they needed to use the latest shiny thing in cars or fridges or ACs or whatever.

[doubled112]

Imagine we could build a dishwasher that didn't throw water on the floor?

Oh well, we got WiFi instead. That's fun, right?

If a dishwasher needs a firmware update, I might simply argue it was defective. Not everything needs to be secure or updated constantly. It shouldn't have network access to begin with.

[talonx]

Exactly. When will vendors stop touting WiFi access as a "feature"?

[ilyt]

And when you upgrade ssl lib now 200 apps are not vulnerable to exploit, vs updating 200 different containers

[interactivecode]

unless the update requires version updates on multiple services which means the versioning and rollback has the same effect as a more monolithic codebase. but with the added complexity of not knowing how it all impacts each other.

In my experience updates to smaller services are often trivial, or for updates that are actually impactful it would be way easier to coordinate in a monolithic codebase.

[steveBK123]

yes my first job over 15 years ago was essentially micro services, and we ran into this type of problem..

trivial updates to individual services could be iterated extremely quickly

systematic changes to behavior across services were so hard they became incredibly uncommon

[bluGill]

Not everything is developed by the single vendor. Sometimes they buy a program from a vendor they don't fully trust.

There is also security, I don't care if someone hacks my radio system nearly as much if someone hacks the brake system. Containers is one part of the total package to isolate parts so if there is a hack the whole system isn't taken out.

Computers are a large system. Someone in "the other group" making a mistake can bring your part of the system down. Much of the code is written in C or C++, and so one "old school" programmer can make a mistake and write past their memory into a data structure you are using, and the bug report goes to you not them.

If you have the above system, when splitting the monolith apart you will discover libfoo.so that both depend on, and the two groups will want to upgrade separately: containers allow this to happen, without modifying your build system to give them different versions or otherwise allow two different libraries with the same name to sit on your system.

The above is what is obvious enough that I can talk about it. (I work on an embedded system at John Deere so I cannot comment more than the above even if you ask.)

[spyremeown]

https://www.toradex.com/torizon they do and it works quite well. Super handy to have CI/CD and easy OTA updates with containers.

[ilyt]

Especially a thing that should be just "a device that puts video stream on whatever bus it uses". infotainment already uses html/js based UIs, just embed video player in there...