[superfrank]

- Expecting engineers to always write perfect code is insane. Mistakes happen.

- If bad code makes it into production, that is a systemic failure not an individual one (Why didn't the bug get caught in code review, QA, etc.)

- No one is going to want to work on a project where a single failure can taint their career.

- What if I use a 3rd party lib and that is where the bug is. Who is at fault then? What if the code isn't buggy, but I'm using it in an unexpected way because of a miscommunication? If I am only allowed use code that I (or someone certified has written) development is going to move at a snails pace.

- What if I consult with an engineer who doesn't have a certification on a design decision and the failure is there, who is at fault?

- What if the best engineer on the project makes a mistake and ends up banned? Does he/she leave the project and take all their tribal knowledge with them, or are they still allowed to consult? If they can consult, what stops them from developing by proxy by telling other engineers what to write?

Not to be a dick, but this is an awful idea that would basically kill the self driving car.

[davidgay]

> - Expecting engineers to always write perfect code is insane. Mistakes happen.

In safety-critical fields, setting a much higher quality bar than the regular 'it seems to works, the tests pass' seems perfectly rational to me. We can now write provably-correct C compilers (CompCert) and OS kernels (SeL4). There's no excuse for not putting similar levels of effort[0] into something as safety-critical as self-driving cars.

[0]: Note that I'm not advocating for "provably-correct self-driving car software" (that may not be the right approach, as a formal spec is likely unrealisable), but find the argument that "it's ok to write buggy spreadsheets, so it's ok to write buggy self-driving cars" to be morally unacceptable.

[PinguTS]

There is ISO26262 for automotive about safety.

Yes, there are reviews, QA, and all of that. So, yes, there is no single person responsible (exceptions apply).

But there is no excuse for using 3rd party libs. Just don't use it. If you not know: do not use it.

That is the reason, why certifications are for. The same rules apply for medical and other areas.

[colatkinson]

> But there is no excuse for using 3rd party libs. Just don't use it. If you not know: do not use it.

Wait, what? That goes against one of the core benefits of open source software--that having many eyes on a problem decreases the risk of bugs. I'm willing to bet that if Uber had to implement their own machine learning/vision libraries from the ground up, there would be significantly more issues.

[henrikeh]

Is there any evidence that this is the case? That simply putting more eyes on a system will reveal its problems?

Certification etc is about process. Open source code can be used in a safety critical product, but it must audited and confirmed against the system requirements.

[PinguTS]

Please do not get me started with the argument over "it is open source, so there are many eyes who have seen the code".

The problem with that is, nobody audits code, if it is working just for fun. And even if it is buggy, then most people look for bugs in their own software and then they work around, so that the original piece is not modified.

We have this seen in many open source projects. Remember all the obvious, mostly security related, bugs that weren't uncovered for years. They weren't uncovered because everybody thought: "huh, that is hard. I assume that other more experienced than I will have reviewed it, so I will trust it."

The thing with certification is, that it is required that it is really reviewed. That there is a guarantee that it is reviewed. That there were people with a different mind set, with different background have reviewed it and as such have brought in their own view.

Certification does not guarantee that something bug free. It guarantees only, it is reviewed. Open source has no guarantee that it is reviewed. There is only hope, that someone has reviewed it.

[Schwolop]

In ISO13485 medical grade software (certain levels of it anyway), the same concept applies. Anything not written in house is "SOUP"; Software of Unknown Provenance. You're required to pass that through a review process before using it, and in many instances it's not worth the effort to review compared to instead just re-writing it yourself.

[superfrank]

> But there is no excuse for using 3rd party libs. Just don't use it. If you not know: do not use it.

Pretty much everything in development relies on the work of other people. I used 3rd party lib just as an example, but what if it's in the framework or even the language that an engineer uses, who would be at fault then? You can't expect every developer to have gone through the entire source code for whatever language they are writing in.

Sciences build on each other and and after a certain period of time you have to take things for granted in order to keep moving forward.

> The same rules apply for medical and other areas.

No, they don't. Doctors kill patients all the time and they aren't banned from medicine for it. There is an investigation, they make sure it wasn't intentional and there wasn't any gross negligence and that this isn't a repeating pattern, if none of those are the case they see what they can learn from it and move forward in hopes that what they learn can help other doctors.

[Glawen]

That's why in iso 26262 you can only use certified compiler and tools. Rust compiler is not certified for example. Sure you can use 3rd party libs, but they must be certified.

I agree dith parent: never use copy paste from internet in safety critical SW, anyway it most probably isn't designed for your use cade. Personally I always have been disapointed by copy pasting stuff, it was always buggy somehow. In the end I always reimplemented it from scratch by reading the theory

[velobro]

Structural engineers are liable if the building they design collapses. I don't see why software engineers in safety-critical fields should be any different.

"The engineers employed by Jack D. Gillum and Associates who had "approved" the final drawings were found culpable of gross negligence, misconduct and unprofessional conduct in the practice of engineering by the Missouri Board of Architects, Professional Engineers, and Land Surveyors. Even though they were acquitted of all crimes that they were initially charged with, they all lost their respective engineering licenses in the states of Missouri, Kansas and Texas and their membership with ASCE.[22] Although the company of Jack D. Gillum and Associates was discharged of criminal negligence, it lost its license to be an engineering firm." - https://en.m.wikipedia.org/wiki/Hyatt_Regency_walkway_collap...

[simion314]

I think the company must pay a lot if a failure happens, if NASA makes a mistake that will costs so they make sure to think hard, have good processes and do a lot of testing.

From what we seen so far the Uber car failed to detect an obstacle, we also had Tesla crash where the car did not seen a truck so it is obvious that there are some major issues that are not tested for. They need to have better tests and maybe get better security drivers inside the cars so they don't text on the phone on the job.

[chopin]

So you can't be bothered to vet 3rd party software for safety critical uses? That sounds like a no-brainer to me. You can't just slap some NPM libraries together for that use case.

[InclinedPlane]

This is a straw man. Nobody is asking for perfection. What is needed are processes and regulations that ensure a high confidence that companies with self-driving cars have done the best that can be expected of them to ensure their vehicles are safe and that they have met a minimum bar in that regard as well. There are many industries where this already happens, there's no reason to single out self-driving automobiles as being impossible to regulate properly.

[always_good]

I can agree with this, but if a company wants to put self-driving cars on the street, I don't see how certificated engineers on their staff is the thing that matters.

Does that demonstrate that the cars are safe? Even a little bit?

To me it just demonstrates that a grunt is on the chopping block for what amounts to systemic failure.