[cabernal]

This and the John Deere bug posted earlier make me a bit concerned over the accumulating evidence of unreliable software ruining people's lives...

What can be done? Mandatory audits, pen testing?

If this is an organizational problem, more vacation? limiting overtime? rethinking employee incentives?

[danpalmer]

Pentesting and auditing aren't great solutions here. They can be useful on small scopes but a big system like this, it's unlikely to be hugely impactful – it will find things, but who knows if it finds enough.

In the UK in the wake of the 2008 banking crisis, a number of positions in banks became criminally liable for issues under them. If you're director-level or above (I think?) then you may be ultimately put in prison for negligence or issues like that which occur in your department. This is rare, not sure if it's been used yet, but it effected a cultural change in consumer banking as a bunch of execs suddenly had their necks on the line if someone under them did something wrong. I don't believe this is too hard-line in practice, I think a defence is "look at all these reasonable steps we take, we couldn't have foreseen this", but it had the impact (source, a good friend of mine is bordering on this level in a UK bank).

I wonder if a similar thing could work in a wider way across more industries - not with the intention of criminally punishing lots of people, but with the aim to change the culture around responsibility to the public and other stakeholders in the work that we do.

[viraptor]

> What can be done?

Not taking software results as a fact. Software report stating X in court should be equivalent to "the person who wrote this in a hurry would say X, but it's not a sworn testimony".

We should have the person presenting any report like that be personally responsible for the contents. If they aren't willing, it shouldn't be presented.

[Silhouette]

We should have the person presenting any report like that be personally responsible for the contents. If they aren't willing, it shouldn't be presented.

I don't think making it personal works at scale. You can't reasonably expect everyone giving evidence in court, say every individual police officer who is a witness to a speeding offence, to be a technical expert on the technological tools they are given to do their job.

Instead, as you implied in the previous paragraph, the weight given to any evidence derived from technology should be proportionate to the credibility of that technology. If it's a device that has to be vetted and approved according to strict regulatory standards and in court there are two other concurring sources of evidence, that's clearly a much stronger case than a single reading from a single device whose calibration has reasonably been called into question at trial that is being presented as the only evidence in that trial.

[viraptor]

> say every individual police officer who is a witness to a speeding offence, to be a technical expert on the technological tools they are given to do their job.

That's what I was going for. If the officer doesn't understand the limitations of their tool, they shouldn't testify in court beyond "I pointed it that way and read the number, as trained".

There are existing cases where the speed reading is contested because the handheld speed cameras can move slightly and bounce first off the side mirror then off the reg plate giving you "extra speed".

My point was that if you say "that person was speeding" you should be responsible for that statement afterwards, but you can say "I used the provided tool and got reading X", at least the doubt is there.

[Silhouette]

FWIW, I'm reasonably sure that's exactly what does normally happen in that particular case. Police officers sometimes speak in a slightly stilted way in court here in the UK, partly because they use words carefully chosen to be statements of fact as they know them and not to draw conclusions that are a matter for the court to decide.

[icegreentea2]

It's not about positive incentives, it's about the lack of negative incentives. More true negative incentives need to be shifted onto the production side, back onto the corporations, its officers, its middle management, and if required down to the individual contributor.

Corporate structure helps diffuse and deflect responsibility. Each group (executive leadership, middle management, and ICs) gets to diffuse and deflect responsibility and liability onto each other.

We already have all the positive incentives in the world - cash money. It's not enough.

[Chris2048]

Standards. Just say certain things, payment systems, need to meet certain levels of auditability (does it record all relevant data, and can I see them after the fact), verification (is the data correct and can I prove that) and privacy.