[qasar]

Qasar here (x-YC founder and partner). So im building a simulation company. Not commenting on Uber specifically but the reason lots of companies don't have great efforts is simulation is actually really hard. It’s not building a web application (tested methodologies, established best practices, lots of experienced engineers in the space...etc).

You can get a lot done (and get wonderful demos) through research vehicles. The issue with sim is similar to the issues with AVs as a whole - getting to 90% is much easier than getting to 99.9999%.

Also, there is this weird thing in AV (and specifically in-house sim) where pointy-haired managers think "if we put 30 SWEs on the problem - we'll make progress!". While raw eng horsepower matters in some software problems, when you are on the line between research and engineering, more SWEs might actually make getting a robust product up harder.

Another thing we've seen is in-house efforts overfit solutions. They don't really have a wide viewpoint (nature of how secretive the space is), and therefore go down paths which are just plain wrong but no one is there to challenge them.

[solatic]

Is this really so surprising?

Nobody (well, theoretically someone, I'd like to be proven wrong) has a self-driving algorithm which is formally proven. If we can't get, to use your example, web engineers to formally prove their web applications, when they have all these structural advantages, then how can we expect something unproven (har, har) to be formally proven? So there needs to be an extensive QA test suite.

And for all of these other relatively-mature engineering sectors like web engineering, how many of these companies are really investing in QA? In strong and comprehensive test suites, in security reviews, in performance testing and UX scoring before release? Almost nobody, right? I mean, the only reason why society can get civil and biomedical engineers to submit to strict quality control is through regulation, but no regulatory body has any real idea of how to apply that to software. And autonomous driving systems are supposed to be different?

Mythical Man-Month came out 38 years ago. The Pareto Principle, not so much as being explicitly a 80/20 split but about how the vast majority of the effort is needed to achieve actual completion rather than just a proof of concept, was originally coined in 1896. Suddenly it doesn't apply to Uber?

Is it really so surprising that it's so difficult to find senior practioners in these areas when few companies are committed to that level of quality, and the companies which are tend to be so successful that they can afford to pay compensation at rates so far above the rest of the market so as to ensure that few of their senior engineers ever leave?

[woolvalley]

The reason why there isn't perfect QA for things like webapps is a cost reason, not a knowledge reason.

Society usually doesn't need a formally proven cat pics sharing app, and most web apps, including facebook, are that level of application. They would rather pay a cheaper price than to have a perfect expensive thing.

You'll also notice that as organizations get bigger, they do invest more in testing, QA and regression detection.

[speedplane]

A counter example is in the aerospace industry. Most software has to be d-179b certified, which requires not only testing with complete code coverage, but also complete branch coverage. Every possible outcome in every “if” statement must be tested.

Needless to say, this is expensive. I’ve heard estimates that it costs roughly $1000/line of code.

The point is that there are methods to increase reliability, but there isn’t the financial incentive to do so.

[speedplane]

Separate point, I’d argue that small organizations also have quite a bit to gain from a solid testing program.

If there is only a single developer working on a project, that developer can manage much much more code if there are solid tests behind it than if she was just winging it with every commit.

At some point the developer starts to forget the code they write and starts relying on the tests they built long ago.

[qasar]

Agreed. AV regulation is early and we're involved in that conversation. We (and I think many in this business) deeply care about safety and believe better technical understanding among lawmakers around software safety will result in better laws. Re: mythical man-month and pareto - 100% true for all AV makers but theory and practice is different. I imagine both their investors and their perception of market forces drive decisions to some degree. This is where you see some gaps (not always) between traditional software lineage companies and traditional OEMs.

[w_t_payne]

Very interesting perspective.

I always assumed people would integrate an external simulator like CarMaker rather than build their own.

I'd be exceedingly interested in hearing about your product when it comes available (although my interest is in simulation for maritime autonomy rather than automotive).

[qasar]

We're super familiar with CarMaker -- great for dynamics and some level 2 stuff but there are some big difference between autonomy levels (from simulation perspective at least). At higher levels, autonomy algo development is more like modern software development. CarMarker is a fantastic product (along with CarSim, dSpace ASM...etc) but they were built for a different use cases. A very crude analogy would be tools for J2MEE vs modern tools for iOS/Android. Our product is available, just not very public about our more detailed approaches because that is really where our IP is. Feel free to reach out if you want to chat. Obv, i'm super interested in the topic.