[svorakang]

Neither safety now environment is something you can easily wave at like that. Also, you're completely missing security concerns and legislative.

I have worked in the automotive embedded software industry since 2009 and I have got caught in the safety track in my career. It's a strange place to be, because the basics are extremely simple, yet it takes hundreds if not thousands of man-years to get a modern vehicle reasonable safe just in terms of the electrical system (this includes the software in automotive terms). There are so many ways to make a mistake that could easily result in an accident. Even the window regulators have non-trivial implementation concerns for anti-pinch. Allowing a random hacker to override this is a terrible idea. Now imagine what kind of mess you could do with brakes and steering...

Designing a vehicle to be hackable will very likely lead to an unsafe vehicle.

I believe what I just wrote applies similarly for security too.

Furthermore releasing software for the market, extensive testing is carried out by an independent body to ensure that legislation is followed. Even conceivably simple things such as lighting or headbeam alignment is a pretty large problem domain by itself. Also, so is just the communication standards for diagnostics.

I would say that large changes would be required to transform this industry. In some, protected domains there is use of open source, such as Qt/Linux for HMI, but opening the HMI to be fully hackable is unlikely to happen. There is quite some liability to make the HMI non-distracting.

[jacquesm]

Oh dear, I wonder how I'll ever be able to use the code I wrote over the years that controls uncounted lathes, mills, plasmacutters, lasers and a whole raft of other industrial tools.

Obviously the only people that can be trusted with our safety are the manufacturers, because the people whose lives are on the line are irresponsible madmen.

> Designing a vehicle to be hackable will very likely lead to an unsafe vehicle.

Vehicles are hackable, but they're not documented which makes them more dangerous, not less dangerous. Witness comma.ai and others.

[ncts]

> I wonder how I'll ever be able to use the code I wrote over the years that controls uncounted lathes, mills, plasmacutters, lasers and a whole raft of other industrial tools.

You are knowledgeable enough to make them work. Many aren't. Some can't be. Hacking requires knowledge and skill, and most importantly, being contained. Cutting yourself with your self-programmed hackable laser in your garage is unfortunate, but cutting other people is a disaster you can't afford.

> Vehicles are hackable, but they're not documented which makes them more dangerous, not less dangerous. Witness comma.ai and others.

I see two points here.

1. Security through obscurity is bad. That's true, but we have "business" in the play, so that's how it goes. Maybe push for better regulation.

2. comma.ai, an "autopilot", based on reverse engineering, or as you put it, the base product "not documented", thus makes it "more dangerous". No, it's dangerous not because the base product is not documented, but because there's no real autopilot at the moment, and comma.ai is irresponsibly advertising as being able to "drive for hours without driver action". There are many "black box" products with a ToS that forbid reverse engineering. Does that make the product inherently more dangerous too?

Besides, you seem to suggest that, with open products, people can not make things unsafer. That's not true. Some don't know what they are doing when they "hack" things.

[adhesive_wombat]

I'm all for open things, but that's a false equivalence. You don't use those tools on a public road around unsuspecting others.

In the same way you can't just merrily hack about with a plane. The FAA don't really care that much if you die in your experiment. They do care if the burning wreckage falls on someone minding their own business.

[LeifCarrotson]

No, Jacques and I hand those things off to random unskilled laborers who come work at a robot cell or CNC for $15/hr without any real knowledge of the code and safety standards. "Push the green button, trust us, it's safe."

FAA regulations for experimental or kit aircraft are wide open. As are automotive standards, having just helped my buddy get his sand rail DOT certified for use on the road. A tube chassis, '65 Volkswagen rear end and VIN, safety glass and manually-squeezed washer bottle and manually-swung windshield wipers (for the DOT guy, then never used again), un-boosted non-ABS brakes with a cutting brake in the loop, and it's cleared for use on the Interstate if we were crazy enough to do that. It's only the manufacturers who claim they're protecting the public by locking down their designs.

[svorakang]

Road approval looks very differently for mass-produced vehicles. It is not an easy thing to pass.

[jacquesm]

And what makes you think that the current crop of automotive software written in either asm or unsafe C is going to be any better than what you or I would produce? I've had a very recent model Mercedes C-class nearly kill me twice on account of buggy software. So much for that 'stellar' (pun intended) reputation. My current car is as dumb as it possibly could be.

I'd expect that if any ECU software was to be released that we'd finally realize how bad things really are and that there would be a massive amount of work done on making sure these pieces of critical software would be as safe as they could possibly be.

Note that the norm is 'a subset of C deemed to be safe' but that what I've seen of such development would not pass my personal threshold for quality work. In fact, rather the opposite. On the plus side, the hardware people usually know their stuff and realize what is dangerous to pass to the software people so with some luck your vehicle will use an FPGA for any kind of really safety critical stuff (or processors embedded with the relevant hardware, such as ABS and so on).

[formerly_proven]

> I'd expect that if any ECU software was to be released that we'd finally realize how bad things really are and that there would be a massive amount of work done on making sure these pieces of critical software would be as safe as they could possibly be.

toyota/denso michael barr testimony cough

edit: oh, there's even slides now, you don't even have to read the court transcript https://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRU...

[jacquesm]

Yes, that's one of the better known cases. But I've looked at similar stuff under NDA and I'm more than a little bit uncomfortable with some of the stuff that I've seen.

The scariest kind of developer to me are the ones that believe that they and only they are able to create safe software without outside scrutiny of what they produce.

Of course that scrutiny would have exposed emissions cheating software right on day #1.

[intern4tional]

This comment chain appears to have a fundamental misconception of what constitutes safe and what does not.

Automotive standards and automotive coding standards approach safety in a different way than most people think (and given your comments I would say this includes you). If you're curious, you can have a look at some rules to evaluate automotive code that are published here: https://github.com/github/codeql-coding-standards

In short, the rules do not aim to eliminate failure or crashes, but rather make the crash predictable and uniform when a crash occurs so that it can be dealt with. This is further complicated by where and how the automotive manufacture chooses to implement safety controls. It is entirely possible to have a bunch of unsafe code running somewhere on a car, and simply have a small safety shim around said code that prevents the unsafe code from impacting the safe operation of the vehicle.

With that in mind, let's take the example that you use here of emissions cheating software. Emissions is likely not considered safety relevant (it might not even be QM, it just might be some code) and so no safety requirement applies to it. So, no real scrutiny would happen there regardless, at least from a safety perspective. See, validating that software passes a particular safety certification is time and money intensive and manufacturers therefore keep the amount of code that they qualify as safe to a minimum. This means as an example that the infotainment systems of many manufacturers are not safety relevant and no safety function should exist on or interact with them.

A few other things to consider from other threads:

- Telsa doesn't necessarily follow or adhere to safety standards. They (Telsa) are explicitly non-compliant in some cases, and this is partially why there are investigations into their practices.

- Industrial robotics code is just as bad if not worse than most automotive software from what I've seen. As you note, its that these robots are not under manual control

- None of this prevents the software from being open source. There are plenty of safety qualified open source projects. This simply limits who can contribute and how contributions are managed. The main reason why many things in automotive are not open source is that the ECU manufacturer isn't interested in doing so, and the Tier 1/2/3 that does the implementation is even less so.

[mikrotikker]

I got a friend who was almost killed by traction control. Pulling out of his driveway onto a 3 lane road, and having 2 cars 1 overtaking come around the corner very fast. He tried to punch it to get out of the way but: ”computer says no”. Longest 2 seconds of his life he reckons, waiting for the gas pedal to work again. Both cars had to brake very heavily.

"Safety"

[jacquesm]

Yes. That sort of thing. I had a 2015 model C-class Mercedes make two fairly serious attempts at killing me before I got rid of it. The first time it could have been a glitch, the second was so obviously borked that I still wonder why they released that stuff on the road. And no way to disable that 'feature'. I'd have loved to analyze what exactly went wrong there and why a bridge and an advertisement by the side of the road were classified as imminent frontal collision.

My present car just does what I tell it to, no gizmos. It may be statistically less safe (I'm willing to believe that) but at least it doesn't actively try to kill me in the name of keeping me safe.

[sircastor]

You don’t drive your lathe down the freeway at 80mph. And your lathe probably isn’t surrounded by hundreds of other lathes also going 80mph. And probably not being run by people only half paying attention.

Tell me about the liability laws in place related to you operating your lathe, or the state-required licensing and insurance that each lathe operator holds.

A machining tool is worlds away from a motor vehicle.

[jacquesm]

Control software for a lathe (or, in fact any industrial robot) is far more dangerous than a car. Go visit any metal workshop and take an inventory of the number of people and then count the number of digits. The fact that they don't cause nearly as many accidents is because they are not normally under manual control. But when they are better find some (preferably solid) cover.

And meanwhile, Tesla gets to beta-test their self driving stuff in public and I can't look at the source code to my ECU to figure out whether or not a certain behavior is by design or an indication that something is broken.

[adhesive_wombat]

Cars will do a lot worse than reduce your finger count. In the UK, 68 people died in work-related accidents. Nearly 1700 died on the roads. (The same figures in the US are 5k/40k, go figure).

And I agree that Tesla shouldn't be beta testing on the roads. But neither should anyone else.

[jacquesm]

The main reason for that is that there are a lot more drivers than there are lathe operators, and that lathe operators usually injure themselves.

There is a relatively low chance that you'll find a lathe on a busy intersection or on a freeway during rush hour. I thought that was obvious.

[ahmedfromtunis]

Not necessarily irresponsible madmen; just curious.

Because I bet you if I buy a new car and discover that I can access its internal components via an API, I will be toying with it.

On any other platform that would never be a problem: found a bug? Just restart the container!

But with a car, this might mean a bug in my code manifesting itself while I'm driving 120 kph. And maybe there's a pedestrian crossing the road and I can't stop in time because the bug makes the brake 60% weaker.

This time however, there's not a restart docker button.

I'm sure if this happens people would be attacking Ferreri viciously the way they pile up on Tesla whenever a douche sleeps at the wheel going 100 kph, even though the company said before that that's not safe.

[jacquesm]

Then we should all go back to security-by-obscurity and trust in the man behind the curtain for computer security as well. And we all know that that doesn't work, so why is there this conviction that the embedded programmers at car companies are made from magic?

It's precisely because cars are so dangerous that the code should be open to scrutiny. And of course - at least in the past - the argument has been made that more eyes do not make the bugs more shallow, but in practice if there is an incentive (such as personal safety) people will expend a lot of effort to figure out why stuff goes wrong.

What it would do is to take away any kind of excuse that manufacturers have in those cases where their gear is suspect to claim that their wares are perfect and that it must have been user error. Because I can pretty much guarantee you that if you were to inspect your average automotive code-base that you'd find errors, and not just minor ones. From accidental erroneous emergency braking, untended acceleration to outright malicious ones such as planned obsolescence drivers, emission controls defeat code and so on.

[adhesive_wombat]

Open to scrutiny, absolutely. Anything safety-critical should be freely available to those it can harm. Cars, trains, planes, nuclear reactors, lathes, the lot. I hope your code and schematics is fully provided to worker relying on it being correct. I indeed don't have faith in regulators auditing it properly.

That said I still don't want someone to plonk some GitHub code into the brake controllers, take it for a spin and turn me and mine into meat salsa.

On private land, surrounded by informed and consenting people, sure, go nuts.

[jacquesm]

> That said I still don't want someone to plonk some GitHub code into the brake controllers, take it for a spin and turn me and mine into meat salsa.

The chances of that happening, versus brake fluid contamination, bad lines, seized rotors, rusted rotors, rotors and pads with grease on them and a thousand other mechanical failures are nil. Because brake controllers are always backed up by a mechanical system and the worst thing about a brake is that it could fail.

The bigger problem is that manufacturers that could barely create functional entertainment systems are now actually creating software and hardware combos that can override driver input to the steering wheel and the brakes and in my own experience they are absolutely not qualified to do this. Car software is crap, you can take my word on that. Very, very few manufacturers have software as a core competency.

[elzbardico]

Please define car software.

You have user facing functions, and you have engine control functions, ABS, transmission, and so on.

The first one, I agree, is generally crap.

For the second one, in a lot of models, your manufacturer haven't even written the code, because they buy it from some OEM manufacturer like Bosch.

And I am pretty sure that Bosch is pretty good at writing this kind of software.

[mikrotikker]

Why do we even need computers in cars. The woz said never trust a computer you can't throw out a window. The only computer in my car is the radio.

It's just excessive consumerism and marketing crap. It's not needed.

[jacquesm]

The initial reason was emissions, but now that we have them all kinds of other stuff gets tacked on.

[debatem1]

> I believe what I just wrote applies similarly for security too.

Automotive security is nearly an oxymoron. The reasons for that are simple: the difficulty and expense of attacking a vehicle exceeds the bored grad student/curious tinkerer threshold, and the automotive industry has collectively the worst attitude towards security I've ever encountered.

The depressingly predictable result is that third party automotive security testing is a sport reserved for people who are extremely disinterested in disclosing their methods to you, aka the actual attackers.

[bboygravity]

Why would it (legally) be on the car manufacturer if someone hacks his own car and causes an accident because of modifications to the ECU (firmware)?

This doesn't intuitively make sense to me. At the very least there are probably huge differences between countries when it comes to this?

Aside from the fact that some people would likely love to modify their car in every way possible to use it on the racetrack or whatever private property?

[jacquesm]

Or maybe to make it safer.

[frenchie4111]

I think their point / the general FOSS argument is that those 1000 of man-years would be turned into 10000 man-years if these things were open sourced. A similar security concern could be waived at things like openssl, but it seems pretty inarguable that openssl is a net-positive for security.

[svorakang]

I'm all for open access to the code for the sake of safety. On the other hand, I'm completely against hobbyists accidently bypassing a safety mechanism.

Open access, but secure access to software download could make sense, at least for commodity parts.

When it comes to features with competitive advantage, though, I don't see that OEMs or its suppliers have anything to gain.

[jacquesm]

> On the other hand, I'm completely against hobbyists accidently bypassing a safety mechanism.

Accidentally.

Besides that: it should be fairly obvious that hobbyists are not going to 'accidentally bypass a safety mechanism', they can cut their brake lines as well and they don't generally do this. What you'd see is that the aftermarket would finally be able to produce stuff without dealers in between and people with the 'right' kind of tooling (authorized by the manufacturer) to get your replacement to be recognized by the firmware. Because of course absolutely none of this would ever be used to protect the bottom line. Right?

Also: if anything open sourcing this stuff would likely result in more rather than less safe vehicles, maybe at the expense of a couple of embarrassments. Because I have absolutely no illusion about the people working on these systems professionally to be somehow magically better than the ones that work on them for themselves, after all, they have a pretty big stake in the outcome.

Imagine that, working on your car in a safety related way... replacing brakes, steering housing components, linkages, suspension components tires and so on is all at least - if not more - risky than working on software.

FWIW one of those 'safety features' tried to kill me twice and caused me to let go of my recent car and switch to a 1997 issue vehicle that has behaved quite predictable compared to that modern one. Whose 'safety features' could not be disabled.

[lm28469]

> Besides that: it should be fairly obvious that hobbyists are not going to 'accidentally bypass a safety mechanism', they can cut their brake lines as well and they don't generally do this.

I can already picture the YouTube videos on "how to gain 15% hp" explaining you how to "hack" your car with a 1s "it will severely reduce your engine life expectancy" message at the end. Thousands of people would run this patch without thinking twice

Also how would you pass the controls most countries do every other year on cars ? I don't expect people checking my brake pads to know how to review the random piece of code I deployed to my car

[jacquesm]

Well, that's sort of the point: this is already possible, so in that sense nothing would change. Changing the mapping (essentially the amount of fuel injected based on a bunch of parameters) is regularly done by 'tuners' (between quotes because they don't really tune anything, they mostly burn more fuel for questionable gains).

But that's really not what I would care about. I'd like to read that code to figure out what the failure modes are and what might impact my safety in a negative way.

[mikrotikker]

I can go out to my late 80s truck truck right now, undo the lock nut on the fuel screw and wind that sucker up. It'll be a completely different truck. Will I half my fuel range? Sure. Will it be fun? Sure will be. If I care about the engine I'll attach an EGT gauge to ensure I don't melt the alloy head.

[thomastjeffery]

Hackable does not mean crackable. The best security implementations in the world are free software.

I'm not even a tiny bit convinced that making cars hackable would be a detriment to safety. Give me one example of that happening in literally any other sector.

[svorakang]

You might have a point there, but I struggle to find any completely hackable product that is also safety-critical. Some airplane, nuclear reactor or some train, perhaps?

[jacquesm]

Any old car will do.

[DennisP]

And new experimental aircraft, which are owner-built.

[yjftsjthsd-h]

Okay, so for the moment leave aside the safety critical bits (only for a moment) - what's the excuse for not opening up the center console? That generally is already segregated and only handles non critical functions.

[jacquesm]

Center consoles have been used quite successfully as beachheads by hackers to be able to get into more important systems because car manufacturers are typically utterly clueless when it comes to security. So obscurity is a very large part of their security. Of course that doesn't really work with the most motivated parties (car thieves and their captive techies) having a field day with this.

Hyundai and Kia are reportedly so bad that they ended up paying out a large amount of money to compensate owners.

https://www.reuters.com/legal/hyundai-kia-agree-200-million-...

But don't worry, it's been fixed now. Probably.

[yjftsjthsd-h]

If it's security critical, it definitely needs to be FOSS and user patchable; obscurity is not a reasonable strategy.

[jacquesm]

Agreed, but that won't happen until some regulator wises up to this being a way to reduce vehicle theft considerably.

[pests]

> Even the window regulators have non-trivial implementation concerns for anti-pinch

Tesla just got hit by this a few months back. They had to remove the auto roll-up-windows when you walk away after parking. Apparently they didn't have the sensors or hardware to do it safely.

[matheusmoreira]

> Allowing a random hacker to override this is a terrible idea.

It should be a basic right no matter how "terrible" a idea it is. We bought it, we should have full control. Void the warranty or something.

[lm28469]

Many of things "should be" and "ought to be" but we all sat around a table and decided to make a neat little thing called the law because at the end of the day we're still just apes and apes don't always act in the best interest of their peers

[matheusmoreira]

Do you seriously actually believe the government and The Law has your best interests at heart?

[lm28469]

Well no, that's my entire point, it's not about what work best for _me_, it's about what somewhat works when scaled to a whole country or continent. It's flawed in many ways, and corrupted in other ways, but it's better than the alternative

If I really wanted to do what I want whenever I want I'd live in a hut somewhere in the woods. Law is rarely about _individual you_ it's about the society you live in.

Why do I care if you have your seatbelt or your motorcycle helmet ? Because if you get injured you'll cost millions to the community. Same exact principle here, it's part of the social contract, you enjoy a lot of neat things and in returns you give up some other things.

[dack]

I'm thinking through the ethics of this myself, but I think it's a reasonable argument that you can have the freedom to do what you want with things that you own _assuming you don't impact others_. The issue with a dangerous car is that it puts others at danger.

[matheusmoreira]

Responsibility. If I modify my car and that's determined to be the causal factor of an accident, it's on me.

[lm28469]

That'll make the hypothetical parents of the hypothetical kid you just ran over very happy I'm sure. "I fucked up but it's on me!" and they lived happily ever after

[matheusmoreira]

Yes? Society is actually completely fine with that arrangement. Every single year tons of people get drunk, get into accidents, get people actually killed and are held accountable for it. Yet nobody dares to infringe on their freedom to drink. They tried once and it just didn't work out.

This "give up freedom because kids" nonsense is seriously tiresome. I'm not engaging with these arguments anymore.

[jacquesm]

Somebody please think of the children.

Really, come on: you can do much better than this. The hypothetical kid might be the one that was saved because some random hacker figured out why the latest generation radar based cars keep phantom braking. That argument works both ways, and typically what is fixed stays fixed.

And of course none of these hotshot programmers would ever be seen near the following bit of code[1], which probably impacted a lot of actual children and their health in a very direct way:

     // in case of an emissions test ensure that we pass
     // with flying colors

     if (emissions_test_detected()) {
         // we're sooo environmentally conscious
         lean_burn();
     }
     else {
         // roll that coal
         regular_burn();
     }

[1] cribbed from VW/Porsche AG's internal repository ;)

[jacquesm]

Agreed, and that's an argument for open source, not against.

[lo_zamoyski]

And someone could respond "Okay, fine, tampering with the onboard software voids the warranty and shifts responsibility to the tinkerer." But that's a liability issue. The safety concern is still there regardless of who is held responsible. A change that seems innocuous may, in fact, be breaking safety regulations. This is a big deal and a matter of public concern.

[mikrotikker]

If you clutch those pearls any harder they'll turn to dust.