[jng]

In my understanding, those projects are the modern pyramids of software. Built by sheer brute force at an unsustainable cost, only affordable by a select few.

Large amounts of reliable, performant and scalable software will be built on time and on budget at some point in the future, with a cost and effort similar to today's run-of-the-mill software development. This will happen when, thanks to better understanding of the principles, we can create better tools and techniques to do so.

[tjr]

I think sheer brute force would be far less organized. How do you suppose designing and constructing a building according to designs and building codes is a more advanced, less brute-forced activity than building software according to requirements and industry standards?

But avionics-style software engineering need not be an all-or-nothing approach. Elements of it could be introduced into other programming applications for increased robustness. Greenspun wrote up an excellent article on adding external design review to web application development:

http://philip.greenspun.com/software/design-review

Such would not be a heavy burden on a project, and would likely help catch at least the most glaring errors that went unnoticed by the developers.

In any event, I agree that better tools and techniques offer the tantalizing possibility to help all software be more robust, even if it is never more "engineered". Modern languages have, for example, done away with whole categories of bugs that used to plague C programmers (and still do, unfortunately).

[jng]

"Sheer brute force" is probably a bit of an exaggeration. But just a bit. As you describe yourself, the kind of advance from C to more modern languages is a step away from "sheer brute force" and towards reasonable approaches. And only a small step compared to where we have to get before this is engineering.

I'd say a much larger part of all software projects is dysfunctional compared to the same in architectural or civil engineering projects, and I think this situation will greatly improve in the future, thanks to a handful of qualitatively innovative insights providing enormous improvements that we are yet to see.