[LowKarmaAccount]

> The F-35 requires more than 8 million lines of code, compared with about 2 million for the F-16 and less than 1 million for other fourth-generation fighter aircraft

The F-35 is programmed in C++, while the other aircraft are programmed in Ada, which is one of the reasons that there is so much more code. When you have to code, test, and maintain 8 million lines of C++ in an environment where a single bug can be deadly, you should expect delays.

The choice of using C++ to write a system that is designed to never crash and the obvious ignorance of the "mythical man month" does not give a good impression of Lockheed's management.

[jarrett]

For anyone who's unfamiliar with the term "Mythical Man Month," it's the title of a book which argues, amongst other things, that assigning more engineers to a late software project will make it even later.

The book can argue more convincingly and thoroughly than I can, but briefly: When new engineers join a project, they have to spend a lot of time familiarizing themselves with it, so they're not productive for a while. They take time away from the project's original engineers by asking a lot of questions, which actually means progress slows down.

Further, more people means more complexity to manage. There will be more miscommunication, more instances of engineers' work conflicting with other engineers' work, more differences in coding style, etc.. With more than 200 people on a project, I can only imagine the nightmares.

I'm not familiar with the F-35 codebase. But I have a hard time imagining it could benefit from 200+ people working on it simultaneously.

I'm also skeptical about this idea of 24/7 shiftwork. Will each person will have their own little piece of the codebase that nobody else touches? If so, then why have a nightshift? Why not just have everyone work in the day, since it's all in parallel anyway? If not, how in the world can you have engineer A coding a given module on the dayshift, who then hands it off to engineer B on the nightshift? Software is not like laying bricks. You can't just come on the next shift and pick up where the last person left off. You lose all the context they had in their heads while coding, which is absolutely essential.

[john_b]

Frequently aerospace companies will have a limited number of licenses for various pieces of software. C++ as a language obviously doesn't have this limitation, but a limited number of licenses for various custom test suites and third party software (e.g. Matlab) is frequently a bottleneck. If this is a bottleneck, it makes sense to have these licenses in use 24/7.

Purchasing more licenses is always an option, but usually not a quick one.

[TheCondor]

Maybe it's really obvious, I mean it should be, but Lockheed clearly isn't much of a software company.

Office space and third party software licenses mandating a 24 hour work cycle for the most expensive defense project of all time?! Maybe it should not be surprising. Their peers seem to just appropriate rights and constitutional interpretation to meet their needs, can't these guys just appropriate more matlab or whatever compiler it is licenses? Just seems like it is being treated like a manufacturing problem.

This same class of folks who are making these decisions are all contracting for NSA and have access to your data to keep us all safe...

[jarrett]

Good point. It does make me wonder though: Does the time and money cost of purchasing more licenses outweigh the effect on employee morale resulting from night shifts?

[jfb]

Fascinating, thanks for the insight.

[outworlder]

> I'm also skeptical about this idea of 24/7 shiftwork. Will each person will have their own little piece of the codebase that nobody else touches? If so, then why have a nightshift? Why not just have everyone work in the day, since it's all in parallel anyway?

Because that way you can fit more people in the same facility? Otherwise, it will be unused for 8 hours each day (being optimistic).

[jarrett]

So they're running low on office space? That's a bottleneck for the F-35?

[jfb]

Why not? It's not like they can just go down to the local HackerNinjaRockStar DojoSpace and rent some Ikea standing desks. At Apple, for instance, there are sealed buildings and security requirements that impose a cost on expansion above espresso machines and foosball tables -- and they're not required to meet DoD legal standards.

[jarrett]

This is probably true. Still, it seems like a red flag to me that on a project like the F-35, which presumably has adequate funding for office space and other such things, their only solution for providing offices for their own employees is night shifts.

There's a very good reason not to have engineers work nights. While some are undoubtedly night owls, many (most?) will have lives and families which will suffer from night shifts. That accelerates burnout and increases resentment--both of which are usually running high in a troubled project anyway.

[jfb]

Oh, the project seems certain to be a total clusterfuck; it's just not 100% surprising that they could be gated on physical space.

[cobrausn]

The F-35 has multiple variants, one of which is a STOVL (Short Takeoff and Vertical Landing) aircraft with a vertical flight system. The amount of code necessary to deal with that is likely a large part of the difference in LOC, not to mention the more complicated systems for weapons, radar, communications, etc...

In other words, the fact that it is written in C++ tells you little more than the fact that it is written in C++ and not Ada, but that alone will tell you that they probably expanded their pool of potential programmers by a large margin.

[MichaelGG]

>expanded their pool of potential programmers

Is that true? I'd think that the main requirements would be experience, intelligence, security clearance, probably familiarity with avionics.

Any good programmer worth working on such a project should be able to switch languages.

I'd hope it's not like some corporate web site that just needs the programming equivalent of warm bodies.

[HeyLaughingBoy]

"Should" does not imply "wants to."

I could switch to COBOL in probably a couple weeks, but you'd have to pay me multiples of my current salary to do so for any length of time.

[cobrausn]

Considering the number of job postings I've seen where they want N+ years of experience, I'd say it's easier to find that in C++ than Ada if they do that.

[olympus]

> system that is designed to never crash

Oh, they crash. Software on currently fielded jets crash all the time, pilots call them "software anomalies."

Systems are designed to fail softly and reboot quickly (~1 minute) and sometimes do so automatically without physically crashing the jet. Sometimes the pilot won't even notice.

[iyulaev]

sometimes do so automatically without physically crashing the jet

I hope by "sometimes" you mean "almost every time!"

[olympus]

Sorry, bad grammar. I meant that sometimes they reboot automatically. The systems are designed to always reboot without physically crashing the jet, but sometimes the pilot has to command the reboot themselves.

[outworlder]

> Systems are designed to fail softly and reboot quickly (~1 minute) and sometimes do so automatically without physically crashing the jet. Sometimes the pilot won't even notice.

If that's the case, why not use something like Erlang?

[skittles]

Because you don't want to be in a high performance jet when the VM decides to do garbage collection during a critical maneuver.

[jfb]

I think the difference in LOC count is likely due to the much more complicated avionics in the F-35 than the technology stack, as compared with older aircraft.

[hosay123]

> a system that is designed to never crash

I'm pretty sure no self-respecting engineer would ever attempt to design such a thing (although many might be deluded into thinking that e.g. advanced type systems might help achieve that goal).

> choice of using C++

Almost certainly not a good indicator.. much of those LOC are likely integrations from other projects. Who is to say the alternative didn't involve rewriting some more sensitive, costly-to-produce code in Ada?

[DanBC]

This website has some information about serious bugs. There are a variety of languages being used, by different industries, with either expensive war machines or expensive civil aeroplanes full of passengers or impossible to service space-craft.

(http://www5.in.tum.de/~huckle/bugse.html)

[johnchristopher]

Why did they switch from ADA to c++ ?

[smackfu]

Maybe all the ADA programmers retired.