[silverpikezero]

This is an absolutely fascinating slide set. Thanks submitter very much for the link. I have written embedded C before, and the following facts just blow my mind:

1. The Throttle Angle function in the Toyota code had a McCabe Cyclomatic Complexity of 146 (over 50 is considered untestable according to slides) [slide 38]

2. The main throttle function was 1300 lines long, and had no directed tests. [slide 38]

3. I find the static analysis results quite alarming. [slide 37]

4. 80+% of variables were declared as global. [slide 40]

I find this to be a stunning lapse of quality, especially for a safety-critical system.

[PhantomGremlin]

Maybe they had a "hardware guy" with little software training or experience write the code?

I've done a lot of both hardware and software, and I've seen a lot more bad software done by hardware engineers than I've seen bad hardware done by software engineers. The software guys usually know that they're out of their bailiwick when it comes to hardware design.

E.g. one of the worst in my experience was a 30,000 line shell script, few if any functions, used as part of our production flow. A simple refactoring could have cut it down at least 90%. Even worse, it was totally unsupported because the guy who wrote it was reassigned.

[ScottBurson]

I think this is probably the essence of it. Hardware people see software as just hardware that can be easily reconfigured. That was a reasonable model back in the 8048 days, when one only had 1kB of ROM and 256B of RAM to work in. But these programs are obviously far larger, with orders of magnitude more data paths and astronomically more possible states. Different techniques are required. Someone from a hardware background isn't going to understand that.

[aceperry]

"Hardware people see software as just hardware that can be easily reconfigured"

Not really, not if you have a realistic view of software. When I was an EE student, about half of my classmates loved software, the other half didn't like it and weren't good at it. And in silicon valley, a lot of EEs ended up doing software at all levels. I've always been jealous of how creative software could be, and didn't have the same limitations as hardware. Lots of hardware is usually programmed at low levels by EEs, which essentially ends up being drivers, and other board support software. They also realize that their training and education is not at the same level as software engineers in software.

BTW, it used to be that if you couldn't cut it in engineering, EE/ME/ChemE, and you wanted to pursue a technical degree, most people would go into computer science. I realize it's not like that today, but that's how it was back in the day.

[MrDom]

That's interesting. I graduated in 2005 from an engineering college and none of the engineers went into Comp Sci if they couldn't cut it in EE/ME/AE/ChemE. They went into "Engineering and Management" instead, which was basically a business degree for engineers.

[aceperry]

The only engineering management program at our school is a graduate degree for engineers with undergrad engineering degrees already. One of the attractions of Comp Sci was the availability of jobs, which was why they tried to complete a traditional engineering program in the first place. It was kind of hilarious to see a frustrated student complaining about one of the simpler classes, but then explain that he was enduring the pain because of a job at the end of the tunnel.

[threeseed]

Static analysis does not show you the complete picture. I have worked on countless horrifically complex, buggy and unmanageable J2EE applications that had perfect cyclomatic complexity et al scores. I have also written perfectly reviewed, manageable and well tested piece of code that didn't.

I very much question the experience of anyone who can refer to a codebase as "spaghetti code" without seeing it and relying solely on static analysis. Software is not that simple or transparent.

[userbinator]

I'm going to guess that the applications you refer to are "simple" at the level of individual functions, but that's only because the complexity has been spread out so much that it becomes difficult to understand the whole. That's exactly why I think cyclomatic complexity and related metrics are of little benefit, and may even be harmful - "refactoring" to reduce "point complexity" can result in increasing the complexity of the whole.

I've always wondered what the cyclomatic complexity of TeX is. While Knuth is a bit of an "edge case", it would be fun to see what static analysers think of his code... complete with copious use of global variables, goto, and very, very long functions. Give someone who has never had any experience with TeX the results and ask them what they think the defect rate would be, then show them the fact that it's one of the most bug-free pieces of software ever written.

[sillysaurus3]

it's one of the most bug-free pieces of software ever written.

Not doubting you, but do you have a source to demonstrate that claim? If I make the claim to someone else, it'd be easier to provide evidence than for me to handwave.

TeX version is 3.14159265, so some of those are probably bugfixes.

EDIT: Um. Look, I rarely complain about downvotes, but what's up with the downvoting on HN lately? Is it me, or what? This is a simple request for more information about something I don't know about. It's not an easy thing to Google. It's up to the parent to provide evidence.

https://www.google.com/search?q=tex+bug+free shows a lot of evidence that TeX is absent of bugs, but that's not the question. The question is the total bugs that have been fixed since it was first written relative to every other major software project. That's not so easy to answer. https://www.google.com/search?q=low+total+bug+count brings up nothing relevant. In fact, it could turn out to be entirely false that TeX had a low total bugcount over its history relative to its size, especially during its very days. We don't know, because no one has provided evidence one way or another.

All of this is exceedingly obvious, and it's getting tedious to type out huge edits like this whenever something straightforward is downvoted.

I'm seriously tempted to create my own community at this point out of desperation, one that focuses on technical merit and being nice rather than posturing. I wonder if one already exists? I've heard some pretty good things about newsgroups, but haven't really looked into any.

[userbinator]

From http://en.wikipedia.org/wiki/TeX#Development

Knuth has kept a very detailed log of all the bugs he has corrected and changes he has made in the program since 1982; as of 2008, the list contains 427 entries, not including the version modification that should be done after his death as the final change in TeX.

The file is called "tex82.bug": http://mirrors.rit.edu/CTAN/systems/knuth/dist/errata/tex82....

Wikipedia is (slightly) out of date since there's 428 currently in the above file, but note that not all of these are actual (functionality-breaking) bugs, just changes; for example, #2 is just a renaming of variables and #425 is an optimisation.

That would be 428 total changes, in a span of a little over 32 years, with the majority of them extremely early in TeX's history - #214 was in 1983, #321 in 1985, #400 in 1991, #420 (a "missing goto") in 2007.

[sillysaurus3]

Thanks! That's one of the coolest changelogs I've ever seen. It's pretty amazing to see 9000 lines represent 32 years of changes.

[jacquesm]

There are a lot of people that would be very happy to receive a check from Knuth for reporting a bug (those checks are rarely if ever cashed, for obvious reasons).

So it's not like people aren't looking for bugs.

[eropple]

> what's up with the downvoting on HN lately

I've noticed a rash of downvotes on posts coming all at once lately. As if somebody gets a bug somewhere unpleasant and goes to go downvote all of that person's posts that they can.

(Which AFAICT doesn't impact karma score, but does downvote all those posts.)

> I'm seriously tempted to create my own community at this point out of desperation, one that focuses on technical merit and being nice rather than posturing.

I'm down. Email's in my profile if you want to chat about the idea.

[jacquesm]

FWIW I think you're an asset here and don't let the downvotes get to you. It happens.

[jacquesm]

Did you miss the joke that that release is 'Pi'?

[sillysaurus3]

I'm aware how the TeX version numbering scheme works. Would you please stop making assumptions about me, from our earlier conversation to this one? I was asking about the total bug count of the system since its inception, and 3.14159265 is at least 8 changes since version 3.

I'll skip the sarcasm and say that yes I realize 3.14159265 is Pi.

[CountSessine]

I have also written perfectly reviewed, manageable and well tested piece of code that didn't.

Ok, so I'll bite. The fact that not all bad code scores poorly on cyclomatic complexity scores does not imply that code that scores poorly in cyclomatic complexity isn't bad code.

So you wrote a function with cyclomatic complexity greater than some accepted value (50?) and you claim that it's well tested. I'm skeptical. How many of those code paths did you follow in your tests? What was the ratio of lines of logic code to lines of test code for that function? Can you really say that your tests provided adequate coverage of this function?

I've fixed a lot of cowboy-code with horrible cyclomatic complexity and no state separation, and this report about the 1500-line throttle control function dipping into system-global pool of universally-accessible variables sent shivers down my spine.

[wyager]

>80+% of variables were declared as global

This is actually important for safety-critical programming. It lessens the likelihood of running out of stack space, and it's useful for eliminating dynamic allocation.

Some safety-critical software has 100% global variables, without even using a stack.

[ufo]

But even then, you want to declare those variables as static or otherwise find some way to reduce their scope. Most global variables in the Toyota code were visible and writable from the whole program.

This is not even counting all the variables declared with the wrong type or more than once, uninitialized variables, etc.

[jacquesm]

I think this is possibly one area where functional programming could shine if the garbage collection issue could be dealt with once and for all (and that's a hard one). It's for practical reasons impossible to test something with that much state exhaustively, but once broken up into functional units you just might get there. Something along the lines of Erlang for embedded systems.

[mattmcknight]

No. Functional programming does not magically free you from "state". The world has state, the unlimited number of real world inputs are the state of the program, merely expressed as function parameters. The CPU itself has registers, and stores a call stack.

The abstraction of functional programming does not alleviate all of these issues, and can introduce other ones.

[jacquesm]

Of course it doesn't free you from state. But it can help to put the state into a more manageable context. Just like side-effect free functions can help you with that.

I've done enough embedded programming (and repairs on embedded programming projects) to know just how bad the spaghetti can get and it really wouldn't hurt to borrow a few leaves from the functional world in those cases.

[magila]

I'm not going to defend the other points, but most variables being global is not at all unusual for code like this. These sorts of realtime embedded applications typically have no heap allocation, so the only way to define persistent storage is as statically allocated globals. Modularization in these systems is done by convention, e.g. by limiting which include files are visible to a compilation unit.

[jacquesgt]

Non-stack variables can be static instead of global. Either file-scoped or function-scoped.

[jwilliams]

Exactly.

This is also broken down on slide 40 - local static and file static variables.

[DangerousPie]

It's strange that the slides don't seem to mention this point at all, even though the author is presumably an expert in this field. Why would he make such a big point of this if it's commonly done in the field?

[threeseed]

There is a difference between being in an expert in the theory of software development and being an expert in the reality.

The theory says that bad software is written when you don't have 100% test coverage, static analysis on checkins, peer review, pair programming, the usual guidelines around encapsulation, inheritance etc. The reality is that bad software mainly comes about through poorly thought out requirements, bad initial architecture decisions and ridiculous time constraints.

It's always a shame when you see stories like this that those aspects never really get investigated or analysed.

[nsnick]

Koopman was one of the main witnesses for the prosecution. He has also worked in the nuclear navy and industry specifically designing safety critical systems.

[eropple]

No ECC in RAM, either. Yiiiikes.

Edit: or peer review, or even a bug tracker. My coworkers working on web apps have a better SDLC than these jokers.

[threeseed]

They did have peer review just not on every module. This is very common even amongst companies who say they do peer review.

And curious about the lack of a bug tracking software. Many companies including the one I work for "technically" don't use them. I wouldn't call Rally or Trello a bug tracker but that do work just fine for that purpose.

I also did find it amusing that they called out lack of formal specifications as an issue given that Agile encourages you not to create them.

[eropple]

> They did have peer review just not on every module. This is very common even amongst companies who say they do peer review.

This is not common at all in safety-critical applications.

> I also did find it amusing that they called out lack of formal specifications as an issue given that Agile encourages you not to create them.

This shouldn't be amusing if you understand Agile. Agile is for systems where you don't yet fully know the scope or the desired end behavior. The guts of a production car are as waterfall as it possibly can get.

[sillysaurus3]

It's not acceptable for a system that can kill you to be programmed without at least two programmers looking over every line of code. Common practice or not, given what we know about programming, it's immoral.