As somebody currently working at an automaker on software systems, the amazing thing to me is that a mess up of this level doesn’t happen weekly. It’s rough out here.
Thank you. At least you're honest about it, the other day someone was trying real hard to convince me that software developers at automakers are made of magic fairy dust.
Relatively crappy pay, complex toolchains, long build times, layer upon layer of (really bad) legacy code, badly specified (if they're specified) protocols between subsystems, subsystems that are completely opaque (no source code provided), homegrown OS's or older RTOS's, subset-of-C to keep it safe(r), tricky debugging environments and if you're really unlucky anemic hardware.
I hope I didn't miss anything but I wouldn't be surprised if I did.
Yeah, I think you missed something. The "software architects, heavy enterprisey tooling, and minions" approach to development where some of the architects could be good developers, but they don't develop, and the minions are often not that good and also not given any autonomy, so they are in a state of learned helplessness and just do what they're told without much thinking or initiative. It results in over-abstracted, over-complicated, slow, unreliable, and sometimes just stupid code.
Automotive varies widely between "basically modern Linux systems with proper updates" and the most janky, home-grown update systems imaginable, sometimes even within the same components and teams.
Yah, I know from friends at ford and vw that there's still vxworks and qnx, but even there, good grief, a-b with confirmed boot is about as basic as you can get.
I confess I've seen incredible sloppiness about when a confirmation is done (too early, including in the initial init stages which is way too soon) and watchdogs (spawn off a process that has a while loop stroking the wd - just absolutely pointless).
I've heard all of the above, often "stroking". I never used those because I like systems where you have a random challenge code to respond to. Then the software has to not be acting as wonky to react correctly.
Indeed, a good RTOS from 10-20 years ago works just as good now as it did back then. The only things that change are dev environments and the driver support.