[this_user]

German companies, especially old school industrial ones like VW, have a very hard time understanding open platforms. The view everything through the lense of liability and compliance first. Their thinking is that if someone runs their app on a custom ROM and uses that to manipulate the app in any way, and that causes some extremely hypothetical damage, that they might be held liable for not having prevented this situation.

Obviously, the chances of that are virtually zero. But they'd rather make their product worse than assume with any kind of risk, even if it is virtually zero. That is simply the way in which German enterprises operate.

[leonidasrup]

It looks like the software development at Volkswagen is done by mixed bag of different deparments with different quality.

On one hand you have: Linux at Volkswagen

"Software development without Linux is no longer possible within automotive environment. Therefore Volkswagen Group IT created and maintains a Linux distribution for our developers. This short talk will highlight our starting goal to integrate into the existing environment, highlight our integration problems and solutions with contributing to upstream. Furthermore we will show where Linux desktop need to improve in future iteration to be a good fitting replacement for other systems."

https://media.ccc.de/v/4486-linux-at-volkswagen

On the other hand you have insecure implementation of telemetry: Wir wissen wo dein Auto steht

"Bewegungsdaten von 800.000 E-Autos sowie Kontaktinformationen zu den Besitzern standen ungeschützt im Netz. Sichtbar war, wer wann zu Hause parkt, beim BND oder vor dem Bordell.

Welche Folgen hat es, wenn VW massenhaft Fahrzeug-, Bewegungs- und Diagnosedaten sammelt und den Schlüssel unter die Fußmatte legt?"

https://media.ccc.de/v/38c3-wir-wissen-wo-dein-auto-steht-vo...

[podnami]

I’ve spent time doing software at VW and a few of its subsidiaries, and this matches my experience.

Compliance is everything, and SAFe (Scaled Agile) is deployed as a blunt instrument.

Management treats software exactly like hardware production lines—everything is just an "engineering process" that can be optimized on a spreadsheet.

The underlying assumption is that individual engineering talent is just an interchangeable commodity. Once you view developers as replaceable cogs, outsourcing the entire infrastructure to the lowest bidder in India becomes the logical conclusion.

It’s a textbook case of process-over-people driving institutional tech debt.

[rickdeckard]

> Management treats software exactly like hardware production lines

That's exactly my observation as well. Classic hardware-producing companies have an immense respect on the step of entering mass-production, as whatever issue that slipped through will be multiplied and physically spread across the world.

So they come from the mindset that the dominant mindset is to minimize the SURFACE-area of potential risk. This makes it really hard for them to compete in software-space, because in software the dominant mindset is to just estimate risk.

Neither is wrong, but applied vice-versa is.

- If you treat software like hardware, you end up cutting out everything that could make your product fit more than your decided main use-case.

- If you treat hardware like software, you're placing a bet on behalf of your customer that the product "will be fine", and a (very expensive) bet that this product won't create an aftermath which may destroy your entire company.

Companies which can't manage the distinction here end up putting hardware in the hands of customers they should have built differently and then spend all their resources on software updates just to somehow keep the core function working.

[MichaelZuo]

Why shouldn’t software be treated with the same rigor at Volkswagen scale?

Pretty much all software products typically talked about on HN are laughable at that scale, they have crashes or weird bugs way more often than the six sigma norm of 99.9999% reliability.

For example, I don’t think it’s even possible nowadays to buy a new iPad and use it with default apps and settings for any significant duration continuously. It’s well under 1 million minutes of uptime before failure and a hard restart is needed.

So anything more complex than the simplest possible use case of an iPad is even more of a joke under hardware norms.

[rickdeckard]

> Why shouldn’t software be treated with the same rigor at Volkswagen scale?

No one said that it shouldn't.

What I wrote is, that the approach of minimizing any SURFACE of risk in software creates the (subjectively good and solid) software of previous car-generations (in Volkswagen terms: MIB2 ~ a bit downhill already in MIB3): A solid, predictable and closed product fulfilling its core use-case.

But it DOESN'T create a user experience with those "fun" niche features, competitive remote-access Smartphone features, exposed API's, sudden new features during lifecycle, funny "ludicrous modes" etc.

And today's customers are demanding those features, it's now a hygiene factor for a premium experience on Smartphones as well as on cars.

A Tesla is not considered a "Premium" car because of its premium hardware or manufacturing quality. They disrupted the car-industry by being the first to apply a software-dev mindset to it, and the consumer perceives this as premium.

[LtWorf]

It's considered premium until you see the amount of recalls they do for important things like brakes braking and wheels staying attached to the car.

https://www.go-parts.com/garage/brake-pedal-tesla-cybertruck...

https://www.tparts.com/blogs/tesla-latest-news/tesla-issues-...

https://tesorb.com/cybertruck-rwd-recall-173-wheel-stud/

[anonymousiam]

If they have concerns about the security of their app on some platform, they have the choice to either put "security" into the app, or to trust the platform vendor to provide the security. The correct solution is the first way. Deferring trust to the platform provider is the lazy way.

If their APIs are done correctly, they shouldn't be afraid to expose them.

[okanat]

You're proving the previous commenter's point. VW doesn't want liability. They do not care about "security" just liability.

When they leave the "security" to the platform they can blame them in a lawsuit.

[anonymousiam]

Google has a pretty good legal team. Their developer ToS that absolves them of any sort of liability for anything. So this means VW is just being lazy and not seeking legal protection.

https://play.google/developer-distribution-agreement.html

[klausa]

>Their developer ToS that absolves them of any sort of liability for anything.

This is... obviously not true?

If you could (somehow) meaningfully damage a car via the app, do you think VW wouldn't be liable because of the Google Play developer ToS?

[anonymousiam]

My point was that Google would not be liable. VW cannot evade liability from app misbehavior by using Google Play Protect, so why do they do it?

[lan321]

It will look better for the project lead if there's an issue though. You can say that you enabled everything recommended by Google or w/e, following best practices, and still got pwned instead of arguing that your own security model had a tiny little flaw that no one recognized. And it frees up project hours which can either be the difference between doing the project or not doing it and/or allow you to have other project work billed to this project.

[noisy_boy]

> VW is just being lazy

Maybe they rather be lazy and be able to shift blame, even without much legal recourse.

[rurban]

They don't care about legal recourse. If there's something wrong, they'll just change the laws. That's why they don't care about GrapheneOS users, or any EU regulation which could harm them.

[nightpool]

How else would you build "security" into the app (in the sense of not allowing third-party modifications of it that would open them up to liability), except relying on hardware attestation that the app has not been modified? That attestation necessarily requires the platform provider to be involved.

[plaguuuuuu]

You don't, the app runs on a user-supplied device. They should secure the part that runs on the car and consider the interface between the app and the api to be a user interface.

[nightpool]

I don't disagree with this, but you're completely contradicting the user I'm replying to. I'm pointing out that the user who says "you should put security into the app" is completely misguided or at the very least perfectly explaining why VW is doing this.

[steelframe]

Volkswagon has no jurisdiction over how I manage my fob, which is the client for the vehicle's unlock and start API. Once you hand a bearer token to me that governs full access to the vehicle, including the accelerator and steering wheel, it's not your job to babysit whether I chose to use it while drunk or hand it over to someone else.

[tuhriel]

So you don't have a VW employee coming by your house in the evening to check if the key fob is still in you possession? Sometimes he even does a testdrive to make sure it still works with the car.

Maybe I have to ask that guy some questions....

[nightpool]

Okay, but how does that square with the person I'm replying to's claim that VW should "put security into the app"? You missed the point of my reply.

[Hnedelin]

Except it is their job, that is why certain signals on the car are protected from manipulation. Any attempt to circumvent this and succeeding would require direct action from VW. If they cannot prove that they did everything possible to prevent that, then they are legally liable to the authorities.

Same way that banking apps don’t care if you could screw up your account anyway, they will ban rooted phones just to avoid the risk. Because when something happens, what do you think is more likely? That the customer accepts full responsibility for using a rooted device and says that’s on me? Or that they blame the bank for losing all their savings?

[AnthonyMouse]

> If they cannot prove that they did everything possible to prevent that, then they are legally liable to the authorities.

Laws mostly don't work like that. The seller of gasoline doesn't have to prove they did everything possible to design the product to prevent anyone from using it for arson, nor should they because that's preposterous.

> Because when something happens, what do you think is more likely? That the customer accepts full responsibility for using a rooted device and says that’s on me? Or that they blame the bank for losing all their savings?

You're making the assumption that rooted phones are more likely to be compromised, but it's entirely the opposite. The stock software on phones regularly goes out of support and has known unpatched vulnerabilities (but will still pass Play Protect) and the only way to get a patched system on that device is to install a newer third party ROM. On top of that, GrapheneOS has better security than stock Android even for the same version.

Moreover, that has nothing to do with liability. When the user with the vendor-supplied firmware still gets pwned and has their account drained, they're still going to go to the bank looking to get their money back. All the bank does by going out of their way to block third party firmware is to make that marginally more likely.

[crote]

Easy: separate the systems into the safety-critical ones required for driving, and the nice-to-have ones used for things like entertainment. You can now give the car's owner full access to the latter via all sorts of weird 3rd-party apps, as there's no way for that access to cause serious issues.

They should be doing this anyways, or else you end up with your Jeep being crashed via wifi [0], and having the blast radius of a corrupt album image [1] restricted to infotainment is probably a really good idea too.

[0]: https://www.kaspersky.com/blog/blackhat-jeep-cherokee-hack-e...

[1]: https://www.theregister.com/software/2022/02/10/radio-statio...

[user3939382]

VW didn’t seem too concerned with compliance when they were rigging their pollution tests.

[xenocratus]

They'd have you know they actually cared a bit too much about said compliance itself.

[CuriouslyC]

*appearance of compliance

[Obscurity4340]

The Lady doth comply too much!

[joe_mamba]

Them cheating the tests WAS them ensuring THAT compliance.

In fact, that's how a lot of compliance works in industries where there's little little enforcement and relies a lot on self regulation.

[zie]

That was just engineers engineering their way into creating Electrify America :)

[zelphirkalt]

I am pretty sure that was not the engineers, but someone higher up the food chain ordering people to do that. I might be wrong, but maybe I missed the obvious "/s" or "/i" here.

[zie]

LOL exactly, It was not meant in seriousness :)

Clearly the engineering team didn't know ahead of time that Electrify America would be the end result of dieselgate. Had they known, perhaps they would have been more eager to do the engineering work though! haha

It was just a fun inside joke, since nobody could have assumed the fines would create Electrify America. Personally I'm glad Electrify America exists, though the way it happened was probably not the best path to get here.

EA even has successfully moved on from just being an org forced into existence and are actively trying to take care of customers and produce a good product now that they have some competition.

[SecretDreams]

I think the latter on this one.

[jimmydddd]

Yes, but Hans, that one rogue guy in engineering, did get assigned 100% of the blame from the PR dept.

[this_user]

I mean, the only reason they did it was to be able to comply with the requirements of the test.

But the reality is that every once in a while you have a scandal like this or something like Wirecard, and it happens, because the culture is such that absolutely nobody thinks it possible. That includes officials and regulators whose first instinct will often be to come after the people trying to expose the scandal, as has happened in the case of Wirecard.

[joe_mamba]

>because the culture is such that absolutely nobody thinks it possible

Only naive laymen or newcomers to Germany think it's not possible. German business leaders, lawyers and politicians know exactly how much corruption and scamming is going on in the business sector, and it's not a little.

>first instinct will often be to come after the people trying to expose the scandal, as has happened in the case of Wirecard.

That was purely malicious to try to protect Wirecard, not because the regulators couldn't possibly imagine corruption and law breaking exists, that was the story they used as cover for their corruption.

Like you're a regulator and instead of doing the thing you were hired for and look at the evidence The Economist showed you, you instead "use your instincts" to decide not to do your job and not look into Wirecard because you can't imagine something bad can ever happen? Come on! All those regulators should have been fired and tried for corruption and/or accessory to crime.

[LtWorf]

I think germans are fine with corruption as long as delude themselves greece has more.

[joe_mamba]

Germans have a superiority complex over lesser developed countries like Eastern Europe that they use to justify how their corruption is acceptable, and how having much slower and much more expensive internet than developing countries is also acceptable. They're victims of their own arrogance and lack of self criticism.

[LtWorf]

They aren't alone though. I think the whole DE, NL, DK, SE, NO has the same ideas.

[Hnedelin]

And since they saw what it cost them, they are now VERY concerned with compliance.

[linzhangrun]

“Compliance” only matters when their own interests aren’t involved

[raxxorraxor]

They have a hard time to understand software in general, software developers have a very low standing in German engineering and engineering culture has long been replaced by finance people.

And I don't think the liability is the primary problem, they have a problem with freedom and fear that they lose some mechanism for monetisation. This is why you get subscriptions for heating your arse.

[formerly_proven]

If I had to guess it’s liability concerns around the app-based remote unlock and parking + R155 and CRA. A lot of european companies have moved to require attestation in their apps, likely spurred on by the CRA.

[Perseids]

But why? I'd understand (though not approve) them tightening down everything about the car firmware to the max. They are responsible for the app, sure (it's a "digital element"), but they aren't responsible for the OS the app runs on. The CRA should not be used as an excuse to enact stupid restrictions.

[tadfisher]

Unfortunately, due to the nature of these things, you cannot verify an app is unmodified without also verifying the OS running it is also unmodified. So if VW decides that only their unmodified app may access APIs, then they kind of are stuck verifying the OS.

They can, given basic competence in SW engineering, also verify against GrapheneOS' published release keys. The reason they don't is the same reason Google closed my ticket asking them to include Graphene keys in Play Integrity checks: they don't care.

[microtonal]

The reason they don't is the same reason Google closed my ticket asking them to include Graphene keys in Play Integrity checks: they don't care.

I think the reasons are very different. VW maybe doesn't care. Google does it because it would undermine their stronghold over the platform. If they would allow GrapheneOS, what would block Samsung or another OEM from also sandboxing Play Services and not preinstalling a bunch of Google apps and requesting the same?

This shows why attestation is in the wrong hands. Whether a particular device is attested should be purely based on the security of the device (which would also exclude a bunch of certified devices that Google will happily attest now), not on maintaining a smartphone duopoly.

[crote]

Alternatively: don't add inherently-unsafe functionality which requires attestation in order to have a veneer of "safety".

As media piracy and game cheating has shown: no matter how hard you try, there will always be ways around it. You should assume that 3rd-party device you have zero control over is already compromised, so why not use the API as the boundary layer, stop pretending you can secure the app, and open it up to 3rd-party access like it already is in practice?

[microtonal]

Yeah, in the case at hand it's quite silly anyway. We have a VW car and the primary things you can do with the app is check the charging state, stop charging, and turn on the heating/airco.

Unless I overlook something, the worst attack vector for a compromised phone is: you could drain the battery by repeatedly turning on the airco.

Though I guess they are rolling out phone-based car keys, which may be the incentive.

[vitorbaptistaa]

The same company that supposedly views everything through the lens of liability and compliance does this:

> When the cars were operating under controlled laboratory conditions - which typically involve putting them on a stationary test rig - the device appears to have put the vehicle into a sort of safety mode in which the engine ran below normal power and performance. Once on the road, the engines switched out of this test mode.

>

> The result? The engines emitted nitrogen oxide pollutants up to 40 times above what is allowed in the US.

https://www.bbc.com/news/business-34324772

[4thguy]

The first thing I did when I read the parent comment was to double-check if I mixed-up my German companies.

There's only one reason why they're doing this: it benefits them in some way.

[meyum33]

I guess this mentality makes sense if your products’ failures may lead to actual people dying. And in VW’s case it’s the correct culture working as it is supposed to? My Toyota sometimes feels like it was designed by a lawyer. But I somewhat understand given their history of being badly sued. That being said, at this day and age they probably need to evolve to accommodate some UE principles from the consumer electronics industry. Especially given how cars are getting more computer centric. Hitting a good balance between the old compliance/safety mentality and UE mentality will be hard.

[inigyou]

They have to, because German society is extremely litigious and based on finding loopholes in rules.

You know how some religious groups will string a rope between two houses, count it as a roof as long as they're within a certain horizontal distance of it, so they can follow the rope on occasions when the religion says they have to stay inside, and they think God enjoys them finding these loopholes?

Germany is like that, but with lawsuits. If anyone with money finds a technicality to sue you on, they will. So you have to be extremely liability averse if you want to be successful in business. Also, liability is almost always unlimited. You can be bankrupted by a single bad lawsuit.

[rickdeckard]

No idea what's your source on this, but I see you're spanning quite some "rope" from

a.) a global company in the car-industry being cautious of exposing ANY risk-surface in a product because every issue making it to the field doesn't just bear the risk of very expensive recalls/fines but may also put people's ACTUAL lives in danger, to

b.) the country Germany and its whole society

> If anyone with money finds a technicality to sue you on, they will.

In the car-industry you don't need anyone with money to sue you. If you ship a car which is found to endanger participants of traffic, your company may not recover from the aftermath for years...

[treszkai]

What do you base this information on? I live in Germany and have no idea what you're talking about.

[deanishe]

Same. It doesn't sound at all familiar, but Germans are a relatively litigious bunch (so many have legal insurance). Suing your neighbour because their cigarette smoke wafts into your kitchen. That sort of thing.

But "unlimited liabilities" gives entirely the wrong impression. German courts do not award punitive damages, and fees are generally capped below that amount in dispute.

[inigyou]

Actual damages are unlimited.

[fooker]

It's more about rules than hypothetical liability for Germans.

It's inconceivable that someone would want to use a car outside of it's specified rules.

[like_any_other]

> The view everything through the lense of liability and compliance first.

Wow, so they must really want to avoid the liability of spying after their users and keeping all that data, and to be extra sure to comply with the GDPR, they must keep only the absolute minimum of data, right?

Wrong: https://www.theregister.com/security/2025/01/06/data-describ...

https://dailysecurityreview.com/security-spotlight/volkswage...

When a company behaves as your enemy, don't invent wild justifications how they're actually not. At least leave it to their PR team.

[zrn900]

> Their thinking is that if someone runs their app on a custom ROM and uses that to manipulate the app in any way, and that causes some extremely hypothetical damage, that they might be held liable for not having prevented this situation.

Unfortunately, due to the regulations that have blown up in the EU in the last decade, they are right.

[iamnothere]

I wonder if they would be ok with letting users sign a waiver to gain unrestricted API access.

[neya]

Yeah sure, the company behind Dieselgate and single handedly destroyed the diesel market is worried about compliance? Give me a break.

[bri3d]

Yes? These things directly follow one another: VW are obsessed with letter-of-the-law compliance, so things like end-runs around test routines are obvious solutions.

And VW didn't single-handedly destroy the diesel market; economics and physics did. Almost every other manufacturer was also fudging the tests results in some way. But more importantly, building a passenger car diesel that meets NOx targets doesn't work; by the time a passenger car diesel meets modern NOx targets honestly, the car contains a ludicrous precious metal loading in the catalyst and is only a few percentage points more efficient in terms of consumption and CO2 emissions than a petrol car and the math doesn't add up. Diesel is just not a practical solution for passenger cars; it never was in most ways, but it took the EU a long time to restrict NOx pollution to a sustainable level and expose the physical issues at hand.

[dreamcompiler]

You can have high-mileage diesel cars or low-emissions diesel cars but not both at the same time.

VW knew this but lied to customers and told them they could have both. Dieselgate was their attempt to convince everybody the lie was true.

[Hnedelin]

This was official EU policy, based on french reports, as the french and italian manufacturers actually came up with common rail diesels first. The EU then changed regulations to tax based on CO2, which diesels are better at. They also made diesel taxes lower, to offset the higher prices for diesel cars (often 2000+ euros more than the petrol versions). This was all done in the framework of reducing foreign oil imports.

What VW did was to save money from ThinkBlue systems. Not every manufacturer failed the tests, especially the ones who used exhaust gas treatment did pass, and were more economical than petrol engines. After that, the EU changed emission standards and made them more strict, so VW switched to using dual exhaust treatment. Which made the cars more expensive.

What finally killed diesels was the removal of the tax reduction on diesel fuel. Since it is now taxed the same as petrol, there is no more any advantage that can offset the higher purchasing costs of the cars.

[adrianN]

VW is large enough that different parts of the company can have very different opinions.

[zelphirkalt]

That itself though speaks for a broken company culture. If one part of the company is completely disaligned with the values of good engineering, why should anyone still trust the company as a whole? It seems they at the very least severely lack a good vision then, to uphold the company values or what should be the company values.

[donkers]

That’s how megacorps are. VW has almost 700K employees. Enforcing a company culture on that scale is a very diffuse and difficult thing. If you are evaluating whether you should trust a company based on their ability to enforce values throughout all their orgs, you really shouldn’t trust any company unless it’s a tiny one where this sort of thing can be a lot easier to hold the line on.

[moooo99]

I mean, the app services department doesn't exactly have a track record of perfect compliance (privacy) either, so there is that.

[thyristan]

You don't understand, both comes from the same motivation and way of thinking: You see, compliance in Germany is about pretending to be super-compliant and not getting caught. Everyone will do the dance, make all the moves, and if you seem to make all the moves, you are assumed to be compliant. Supervisory authorities will not really check thoroughly except if you are annoying them or making them look bad. Especially if you are partially state-owned like VW.

In Dieselgate VW got caught, made the supervisory authorities and politicians look bad, which is why the authorities also weren't inclined to sweep it under the rug completely. They just shielded VW from the financial consequences in Germany (German VW customers got shafted).

Blocking GrapheneOS is the useless "pretending" part of compliance. They don't really want to do security, because that would cost money, so they pick some actions that seem drastic, harsh and don't cost them anything to implement. Later, when there is a security incident, they will point to their huge heap of pretend compliance, whine a bit about state sponsored actors, high criminal intent and other obvious deflecting bullshit. But they will get away with it, because they did the compliance dance, so they are obviously compliant and did nothing wrong. Nobody in authority will look twice als long as they are neither annoyed or made to look bad.

tl;dr: compliance in Germany is performative

[hwj]

This even has a name: German Angst.

[hparadiz]

Germans will talk a lot about data privacy but then do stuff like this regularly.

[Loranubi]

One is people one is companies.

[hparadiz]

It sends very weird signals when the EU will fine an American company over some data moving in a direction they don't like while at the same time EU governments will allow home grown companies to de facto force people into using products from those same American companies all while lecturing us about duopolies and privacy only to re-enforce those same problematic patterns. It is absurd.

[ahartmetz]

All countries are more lenient with their own companies. Remember who started grounding the 737 Max? It wasn't the FAA, an otherwise highly respected organization. Who is paranoid about Chinese routers while spying on everyone? Etc.

[mindslight]

No, it's not weird at all. They're each just an outcome of two different regulatory philosophies about how to protect users. We might want both because we're coming from an individualist hacker mindset, and thus see them as similar issues. But this is not how regulatory environments work (unfortunately).

From what I can surmise, the German/EU philosophy is more of a closed world approach - accepting that companies will keep control, then government regulates the companies to stop the companies-with-control from causing harm. If you don't like the harm, your recourse-focus is to petition the government to stop it (eg GDPR). Whereas the US philosophy is more open world - once someone "chooses" to patronize a company, then the company is free to do whatever they want. Your recourse-focus is to stop using that company.

They both have shortcomings and glaring loopholes, of course.