[DanielHB]

I worked in similar systems and you are 100% right. 80% of the time was spent on communication protocols between the different boards and microcontrollers. QAing and solving issues from short-sighted dozens of unique custom protocols that worked in non-standard ways (every time a component needs to talk to another component a new protocol was invented).

When you have dozens of communication lines required between different parts of the system it becomes just as complicated as your average micro-service cloud. Really, a car is a distributed system with dozens of "services". An analogy is that each microcontroller-microcontroller communication use their own custom binary-encoding API that runs on multiple different, incompatible versions of HTTP.

We actually spent considerable amount of time just developing our own custom protocol for communication that could run on all sorts of different physical interfaces (CAN, ethernet, modbus, etc) as well as a series of proxies between devices (so component A can talk to component C through a proxy in component B). And if we had to use a custom protocol from an external manufacturer we had to wrap it into our own custom protocol.

That protocol was actually used for our cloud data reporting as well, so eventually all our data communication would use a single unified protocol from micro-controller to IoT Linux to cloud data-ingestion pipeline to database.

[awongh]

For american cars at least, I read that one of the reasons this process exists is because car companies want to work around union rules for manufacturing by outsourcing components of the cars to subcontractors that they can make deals with.

Ultimately it's a price control strategy to pit these suppliers against each other to lower costs. But it means that designing these electronic sub-systems isn't just a question of the design itself, but also of managing all of these supplier relationships as well, they all have different contracts, you would have to coordinate all of them at once to make sure things are interoperable, etc.

[rconti]

That smells plausible, but from my seat as am armchair car enthusiast, it seems that foreign automakers outsource components just as often.

[ashoeafoot]

and have the same problem with software ? yes and yes

[slipnslider]

>price control strategy to pit these suppliers against each other to lower costs

Apparently that rabbit hole goes super deep in which the large auto manufacturers in the US throw their weight around and force suppliers into selling parts at cost or with razor thin profit margins. And on top of that, they force the suppliers to eat the loss when it comes to cyclical business demand (e.g. storage costs for over-producing during low demand and increased labor costs during times to under producing from high demand)

[datadrivenangel]

Conway's law strikes again!

[jollyllama]

You'd think the overhead of managing the supplier relationships would be more expensive than well-managed vertical integration. I'm guessing it's a failure on the part of admin to count their own costs.

[awongh]

Those employees might not be unionized, though.

[smcin]

If that was a main factor, surely then Mexico should be ground zero for next-generation car electronics design? Like, Tijuana or Ciudad Juarez.

[poulsbohemian]

I can't speak to car electronics design or anything about the capabilities of Tijuana or Ciudad Juarez, but your comment had me reflecting that in the mid/late 90s those of us who were in b-school went from all about Mexico to all about China overnight. With NAFTA et al, American manufacturing was going to be all about developing factories in Mexico and moving good back and forth between the US and Mexico for design and finishing. Then once China joined the WTO it was absolutely an instantaneous pivot to China instead. Maybe all that's old is new again?

[smcin]

Right. The unwritten story of the last three decades is the Mexican people in general have lost out on the huge generational opportunity they theoretically should have from being a (comparatively) low-wage manufacturing and R&D option on the US's doorstep ("friendshoring").

On paper Mexico reaped enormous gains from NAFTA(/MCA) but those went to a small slice of the population, and people on fixed incomes/not in manufacturing/services objectively got worse due to inflation and rising cost-of-living, property prices. The share of GDP effectively lost due to corruption is also an issue.

[chii]

> (every time a component needs to talk to another component a new protocol was invented).

i'm sure that every time this happens, it individually makes sense to do it at the time.

This is a microcosm of how large systems get developed in small pieces, by different people, over a long(-ish) period of time. It's the same in the software world too i think, but presumably has a lot more consolidation than cars (as software for cars might be less common, and thus employees moving between companies is unlikely to make any sort of cross-pollination like there would be for FAANG-like companies).

[pydry]

This makes it sound like the problem is that they either lack a person with architectural responsibility for the cars' electronics as a whole or that person lacks the skills necessary to do their job.

[HeyLaughingBoy]

Bear in mind that all the electronics on a particular car are not specific to that car: there is a lot of reuse across product lines. And there are multiple vendors, each of whom is probably also selling the same, or similar modules to other manufacturers.

[tomaskafka]

No, it’s the org and incentives structure - maybe the only people who have all parts that need to make change under their command are the board, and until now, the software was an unimportant part for them.

[datavirtue]

This aspect of the industry has seriously regressed. We started out trying to standardize and as vehicles have become more dependent on onboard networks manufacturers have gone completely proprietary and have put all information behind lawyers. The consumer is the real loser.

[oarsinsync]

> every time a component needs to talk to another component a new protocol was invented

> We actually spent considerable amount of time just developing our own custom protocol

Not only is this unintentionally hilarious, it’s a real life example of an xkcd comic (https://xkcd.com/927/) that will never cease to be true.

> eventually all our data communication would use a single unified protocol from micro-controller to IoT Linux to cloud data-ingestion pipeline to database.

This, however, is remarkably impressive, that you were able to build a single protocol that fit this end to end use case.

[DanielHB]

It is really hard, especially given you have to optimize for the lowest common denominator. For us it was a 512kb RAM microcontroller, we had to go to procurement to expand it to 2MB RAM and they were not happy about that.

On the other hand it was nice being able to just import a library into your code and JUST SEND A FREAKING MESSAGE without having to deal with thousands of lines of code that were last changed 3 years ago and nobody knows how it works. The scrutiny on the code quality of the common protocol was much higher and therefor much more pleasant to use and troubleshoot.

All the encoders and decoders of messages used the same code in all the parts of the stack (technically 2 implementations, one in Go and one in C)

[HPsquared]

Think of the nightmare 5 years down the road when someone else has to then incorporate this protocol under their own new protocol, with the older ones nested inside.