[jng]

Because one finds many self-taught developers out there who can outperform college-educated compsci majors day in and day out.

Because one can read universally-acknowledged figures explaining how a large number of people with a software engineering degree can't code their way out of a paper bag or pass the most basic "fizz buzz test". (1)

Because we can see non-genius 17-year-olds writing apps that are bought by top tech companies for $30MM, month in and month out. (2)

Because we call it "software engineering", but it still isn't engineering at all. (3)

Software development is still a very, very young field. The fundamentals are not properly understood yet. It will still take decades before they are, possibly over a century. We won't be able to put proper education in place before the fundamentals are well-determined.

Agriculture, livestock breeding and cloth making are many millennia old. Architecture, engineering, and the law are 2-3 thousand years old. Book printing and complex music are about 1,000 years old. Dentistry is centuries old. Cinema is over a century old. We know a lot about how to do those properly, and schools are pretty good at teaching the important parts. Software development is less than 50-years-old, and schools are still dismal at figuring out the important parts (practitioners are only so-so most of the time too). That makes it different.

It would be hard to get more misguided advice than what the OP received (pro tip: don't learn vim, Emacs, configure Linux or switch to Dvorak before you can write functional, working code). That doesn't mean teaching yourself is a bad way to learn.

(1) Why can't programmers program, by Jeff Atwood (http://blog.codinghorror.com/why-cant-programmers-program/)

(2) Summly

(3) Just a random sample, but very representative: I developed a software package for building structure calculation about 20 years ago, helping an architect with the software part. There are manuals enumerating the exact steps to follow and excess safety measures to add: assume 50% extra weight, 40% wind-load force with higher percentages for higher structures, twice the amount of iron to be put into the concrete when certain conditions are met, etc... Those manuals are the law. If you are an architect or an engineer, and you follow those rules, two things happen: (1) you are not legally liable, and (2) the building doesn't fall down! Software projects fall down all the time (read: Obamacare). That is engineering, software projects with today's tools and techniques are not. This will happen some day in software. We are not there yet, by far.

[tjr]

That is engineering, software projects with today's tools and techniques are not. This will happen some day in software. We are not there yet, by far.

Sure we are, at least pretty close.

Commercial avionics software developed to DO-178B standards calls for reams of requirements, verification tests, compliance to process, internal quality reviews, external audits, and sign-off by FAA representatives.

A one-line code change can take days to implement, and might not be released to "users" for months or years.

But the software is extremely robust.

If we wanted to engage in the same level of software engineering for all software, we could. But we don't want to. Developers don't want to, and users don't demand it. If an iPhone game crashes, who cares? If a productivity application crashes, you might have lost an hour's work, but it's probably not so annoying so as to warrant a couple orders of magnitude more cost associated with the software.

But if a software failure could kill people, well, that's different. It's worth spending a huge amount of time to make it perfect.

Avionics software can be so thoroughly tested because it is thoroughly designed up front. You know exactly what it's supposed to do. Much less-critical software is designed in a more ad hoc fashion; or there might not even be a design at all! How much software has been organically grown, starting with an idea and hacking on it until it seemed to work?

If you want to thoroughly test that, you have to go back and thoroughly state what it's supposed to do.

It's quite possible, but by and large it's not desired enough to make it worth actually doing. I'm not sure how this could change, or even if it should change. Instant bug fixes on web applications are cool, even though they come with the risk of having broken something else...

[cbd1984]

> But the software is extremely robust.

Does the specification specify the input as well, or is it actually robust against real input?

By real input I mean stuff like HTML tag soup: There's no single standards document which describes it fully, or even mostly, and it isn't going to be fixed. Ever. It simply has to be processed, to the limits of your ability to process it.

Avionics software is robust, sure, but it's almost a toy problem, its domain is so well-specified. You can ignore so much about reality because you've got a contract which says "We only care about what's listed, everything else can go hang" and in the real world (or, well, in the rest of the real world) you can't usually do that.

[tjr]

A fair observation. In the example of handling HTML input, I would suppose that's not a problem with individual developers, but a problem with the industry. Such a relaxed format should not have been allowed to exist, if the industry cared about its software products being as robust as possible.

I'm failing to think of any avionics application that might handle HTML, but avionics systems do have their own formats to deal with. ARINC 661, for example, is an XML file format for transmitting graphical display elements:

http://en.wikipedia.org/wiki/ARINC_661

Of course, all uses of ARINC 661 data are thoroughly tested. I'm not sure I would go so far as to describe it as a "toy problem", but it certainly does intentionally limit the problem domain to exactly what needs to be dealt with. Malformed ARINC 661 data received would just be discarded, not tried to be displayed in the best possible way even if it wasn't quite right, because that would be unacceptable; the problem would be with whoever was sending the malformed data.

Anyway, you're quite right though; without a precise and unambiguous format definition, you can only go so far down the path of robustness.

[jasonm23]

> Such a relaxed format should not have been allowed to exist.

Amen

[graycat]

Yes, long ago it was observed that the first step in proving software correct was a clear specification of what the software was supposed to do and that for a lot of realistic software writing that spec was unreasonably difficult.

[bstpierre]

> If we wanted to engage in the same level of software engineering for all software, we could. But we don't want to.

Of course. Like building structures, there are different kinds of software. I can build a garden shed by myself as long as I have the skills to get the thing to stand up. If it blows down in a storm I'm the only person with a loss. But just because I can get a garden shed (or even a larger structure like a barn or a house) to survive windstorms without incident doesn't make me a structural engineer.

[jng]

In my understanding, those projects are the modern pyramids of software. Built by sheer brute force at an unsustainable cost, only affordable by a select few.

Large amounts of reliable, performant and scalable software will be built on time and on budget at some point in the future, with a cost and effort similar to today's run-of-the-mill software development. This will happen when, thanks to better understanding of the principles, we can create better tools and techniques to do so.

[tjr]

I think sheer brute force would be far less organized. How do you suppose designing and constructing a building according to designs and building codes is a more advanced, less brute-forced activity than building software according to requirements and industry standards?

But avionics-style software engineering need not be an all-or-nothing approach. Elements of it could be introduced into other programming applications for increased robustness. Greenspun wrote up an excellent article on adding external design review to web application development:

http://philip.greenspun.com/software/design-review

Such would not be a heavy burden on a project, and would likely help catch at least the most glaring errors that went unnoticed by the developers.

In any event, I agree that better tools and techniques offer the tantalizing possibility to help all software be more robust, even if it is never more "engineered". Modern languages have, for example, done away with whole categories of bugs that used to plague C programmers (and still do, unfortunately).

[jng]

"Sheer brute force" is probably a bit of an exaggeration. But just a bit. As you describe yourself, the kind of advance from C to more modern languages is a step away from "sheer brute force" and towards reasonable approaches. And only a small step compared to where we have to get before this is engineering.

I'd say a much larger part of all software projects is dysfunctional compared to the same in architectural or civil engineering projects, and I think this situation will greatly improve in the future, thanks to a handful of qualitatively innovative insights providing enormous improvements that we are yet to see.

[sedachv]

> That is engineering, software projects with today's tools and techniques are not. This will happen some day in software. We are not there yet, by far.

This statement is based on exactly the same fallacy as the featured blog post: ignorance. You are ignoring all the tools and techniques available for software engineering today: garbage collection, lexical closures, Hindley–Milner type systems, purely functional programming, MISRA-C, unit testing code coverage tools, bug tracking systems, distributed version control systems and code review practices around them, etc.

People developing widely used tools are still making idiotic mistakes that had widely known solutions 50 years ago: https://twitter.com/vsedach/status/527904732145537025

Engineering is about learning from previous designs. When you shrug your shoulders and say "software engineering isn't engineering yet" and "the field is too young" you are just discouraging people from learning from past mistakes.

Just because some idiot can pick up an oxy-acetylene torch and cut and weld some metal together doesn't make them an aerospace engineer. What's the difference with PHP developers?

Stop making abstract excuses and start treating software and software practices as tools and techniques. Tools and techniques have a history, a learning curve, and areas for improvement.

[jng]

No, this statement is based on years of learning, practice and reflection, all around the creation and release of many successful software products of many kinds on the cutting edge of software development. I don't know why you assume I ignore all those things when I make my statement.

The list of techniques you provide is a hodgepodge of valuable but limited techniques, inapplicable theoretical results, and voodoo-like superstitions and rituals. They are all insightful and beautiful, but they do nothing to turn the discipline into software engineering. The day software development has turned into an engineering, you can bet the contractors who get awarded big contracts like the Obamacare web site would make sure they spend the $$$ to apply the practices which have been proven to ensure software projects become successful, they are still going to make a ton of money off the top of the contract, and that's what they do for large engineering projects for the most part. Nowadays, they are just at a loss, and try to get by, like everyone else in the industry.

GC: Apple keeps not applying it in iOS but switched to it for a lot of core OS X apps, notably Xcode 5, and switched back to assisted-referece-counting in Xcode 6 and others, since the performance degradation was noticeable. GC is great for many things but not for all. It's a trade-off.

Lexical closures: I am not sure this really makes software work better. I've seen many more well-engineered solid projects in C++ and Java, which lack them, than in javascript.

Hindley-Milner and purely functional programming: beautiful beautiful beautiful Haskell is probably the closest to practical purely functional programming system, and we have yet to see any proof that Haskell makes real software system implementation any better. Don't get me wrong, I love purely functional approaches, I designed and implemented a full custom regular expression engine with a purely functional design and implementation (even if I used C++ to write it), and it's the type of thing where the approach results in much better code in all regards. It's just still not practical for most things. I like to think about Haskell as computation poetry. It's just not practical.

MISRA-C: one entry which is just not compatible with the others. So if you are doing a project using C because it's the only practical option (controllers in vehicle engines being a paradigmatic case), it's good that there is a set of practices that, even if they make software 10x as costly, are quite good at preventing resulting deaths. I don't think this

Unit testing code coverage: another shaman practice, valuable but without providing guarantees. The space of program state is so combinatorially non-linear with regards to the space of its source code (read: entscheidungsproblem) that ensuring 100% line-wise code coverage is just a sad excuse in getting the whole building not to crumble. Better than nothing, still guaranteeing nothing.

Bug tracking systems: more good practices, guaranteeing nothing. Every single released software project I have worked on has been released with a ton of open entries in the bug tracking system. There is about as much team politics as engineering going on in a bug tracking system.

DVCS: yes, git is good and convenient, flexible collaboration, and flexible push/pull-based authority policies allow much more functional team dynamics, including fully auditable "sign-off" as part of the process. Defending this as if it provided any engineering guarantee with regards to the developed software sounds funny. Enforceable, auditable process is helpful and necessary, it ensures nothing with regards to the produced code.

Code review: frankly, this is the pinnacle of shamanism in your list. Four eyes see more than two. Eight eyes totaling 80 years of software creation experience are sure going to benefit the quality of the code. I find it indefensible this as "engineering", no matter if you include it in the commit/merge flow. It's just useful but totally fallible common sense elevated to policy.

Look, I don't shrug my shoulders, and I am not discouraging anybody from learning from past mistakes. I am saying that all the above are a nice but tiny start in the way to this field being an engineering. The difference between you and me is that I am convinced there is a set of tools and techniques which is so much qualitatively better that, once they're available, it will obvious that the current stage is more or less the stone age of software. They will provide a clear, concrete, enumerable set of guarantees about software and its production, allowing us to operate as engineers. Something which none of the entries in the above list do.

[sedachv]

> another shaman practice, valuable but without providing guarantees

This is the same fallacious argument you keep making over and over. You dismiss tools that provide guarantees as "not practical," while calling techniques derived from experience "voodoo."

You obviously have no experience in construction or civil engineering, have never looked at a building code book, and generally have no idea what engineering is:

> The day software development has turned into an engineering, you can bet the contractors who get awarded big contracts like the Obamacare web site would make sure they spend the $$$ to apply the practices which have been proven to ensure software projects become successful

Just like Boston's Big Dig.

> MISRA-C: one entry which is just not compatible with the others. So if you are doing a project using C because it's the only practical option

No one in the real world ever has to retrofit or maintain old buildings and infrastructure projects.

> The space of program state is so combinatorially non-linear with regards to the space of its source code (read: entscheidungsproblem) that ensuring 100% line-wise code coverage is just a sad excuse in getting the whole building not to crumble. Better than nothing, still guaranteeing nothing.

Yeah, and finite element models are a perfect representation of the real world.

> Enforceable, auditable process is helpful and necessary, it ensures nothing with regards to the produced code.

I guess all those CAD data management systems don't help engineers design good buildings either.

> Code review: frankly, this is the pinnacle of shamanism in your list. Four eyes see more than two. Eight eyes totaling 80 years of software creation experience are sure going to benefit the quality of the code. I find it indefensible this as "engineering", no matter if you include it in the commit/merge flow. It's just useful but totally fallible common sense elevated to policy.

Just like structural engineering reviews of architectural plans are obviously useless. No way would a structural review by experienced engineers have helped prevent the Kansas City Regency Hotel disaster.

[jng]

> You obviously have no experience in construction or civil engineering, have never looked at a building code book, and generally have no idea what engineering is:

I have very little experience in a architecture project decades ago, I just described my experience a couple comments above - I haven't claimed any more than that. I don't know what reasonable claim you could have to affirming I have never looked at a building code book - I more than looked at one, implemented in software the rules in there following the direction of an architect, and was generally amazed when I saw how real engineering works. I also spent five years attending an engineering school before working in large "software engineering" projects for 20 years, which I don't think are an instance of engineering at all, but hey.

Enjoy your faith in software engineering, your ad-hominems, your cynicism, and your real engineering career. Good bye.

[thomyorkie]

Software engineering is a relatively young field. However, to say we don't properly understand the fundamentals is absurd.

[jng]

Of course that's your take, definitely not mine. Not easy to discuss though: the obtuseness in today's software building won't be obvious until we discover the principles that turn it into engineering.

In ancient Egypt, pyramid-building must have seemed the unbelievable pinnacle of human achievement, only surpassable by ever higher and larger pyramids. Only now we see the primitiveness of those works (notwithstanding their merit and value).

It is my take that our current level of understanding of software building is similar to the that of architecture when pyramids were built.

Fortunately, advance is now much faster thanks to the many tools that help experimentation and communication, and we should be approaching somewhere reasonable in just a few decades.

[jimmaswell]

In what way does the work of Alan Turing etc. long ago not constitute a firm fundamental understanding of the field?

[jng]

In my view, Turing's results are equivalent in computer science (an important part of software creation) to Pythagoras' theorem in geometry (an important part of architecture): incredibly insightful, fundamental, everlasting and useful. Used directly or indirectly in all early works, respectively, in software and architecture. And only a tiny part of the foundational understanding necessary for really mastering the discipline, a level I think we still haven't reached in software.

[ibebrett]

They are more of an understanding of what theoretically can be computed, rather than anything to do with how to do it or how to efficiently do it. I study lots of theoretical computer science, but it doesn't intersect with software engineering much. Software engineering tends to more relate to project management, operations research etc.

[AnimalMuppet]

Define "understand the fundamentals".

If you mean "the theoretical underpinnings", then, yes, we think we do. Turing machines and Church calculus provide solid theoretical underpinnings, that (as far as we know now) are complete. (Of course, the future could show us that they are incomplete. That's the problem with knowing whether your current understanding is proper - you don't know what gaps the future will reveal.)

However, if you mean "we know how to build programs so they work", there's massive empirical evidence that says that we don't. (That is, we can do it... sometimes. And sometimes we can do it after it takes five times as much money as we expected. And sometimes we can't make it work, ever. And we can't tell up front which will happen.) So in terms of this being engineering - as opposed to computer science - saying that we don't properly understand the fundamentals seems like an entirely reasonable statement.

[jng]

I quite agree with you, but not completely. It's true that Turing & Cburch fundamentals are complete, in that anything we describe in computation can be expressed in terms of those. But I think we are using a very small subset of all Turing-Church based computing, and refinements of this understanding will provide a qualitatively better algebra for the types of software we really create. This will make it easier both for engineers to create software, and for end-users to do things that nowadays only programmers can do.

Both ends you describe (computer science and software engineering) are to me part of the same continuum. We still have articulate most of it with a coherent theory.

[graycat]

A mechanical engineer knows about the properties of the metals, plastics, etc. he uses. A civil engineer knows about strengths of steel I-beams, etc. An electrical engineer knows about capacities of cables, switches, etc. So, these engineers can do real engineering and have transmissions run, electric power networks provide the power, and tall buildings and long bridges stand.

A software engineer knows about the capacity of a server? With what I/O rate, TCP/IP data rate, memory locality of reference, independent threads, memory that can be effectively cached, the memory usage, errors and their consequences?

E.g., one of the problems of doing optimization of assigning programs to servers in a server farm is knowing what (1) the servers can do, also considering other programs the server is running, and (2) what resources the software needs, and it's too tough to know either and, then, in a significantly large and complex server farm, fully to exploit this data for full optimization.

Software engineers are working with at best rubber specs on both the resources they have and the resources they will be using. So, can't really do engineering on the work, e.g., really can guarantee next to nothing. Or, would you want to bet the life of your wife or daughter on some software doing its work without errors and on time? Neither would I!

[seabee]

I agree. Sometimes we do take the reductionist approach, take the Apollo guidance software or seL4 as examples. However, when winging it gets you results that are almost as good in a fraction of the time, you can't be surprised at the prevailing method of software 'engineering'.

[Retric]

Architecture, engineering, and the law are all well over 6,000 years old.

People have changed less (ed: more slowly) than you might think. Consider, people where applying makeup 6,000 years ago and there is some evidence the practice is ~100,000 years old.

[jng]

I just threw an approximate guess of 2-3k years. If it's 6k, more to my reasoning: there's been so much time and effort to learn those that proper principles are now well known and can be taught and applied repeatedly, reliably, affordably.

I don't think people have changed at all, and I didn't say it anywhere above, so I don't understand why you assume I think people have changed significantly.

Our knowledge in a few areas has grown significantly though (read: crafts, science and technology). I don't think anyone reasonable would disagree, but who knows.

[Retric]

No offence intended, I just noticed a lot of those numbers where fairly low. Like where you say dentistry is century's old which is clearly true, but "Remains from the early Harappan periods of the Indus Valley Civilization (c. 3300 BCE) show evidence of teeth having been drilled dating back 9,000 years." http://en.wikipedia.org/wiki/Dentistry

So it's 90+ century's old. Which is one of those, wait what? moments for me that I like to share.

[jng]

That's fine. I don't know the exact details of all those disciplines, I like history, but my knowledge of it is limited. I was within reasonable orders of magnitude of the real dates, or lower, which helps my argument that software engineering is incredibly young, which is the whole point :)

[sedachv]

The earliest known megalithic construction site is 11 thousand years old: http://en.wikipedia.org/wiki/G%C3%B6bekli_Tepe

The problem is that there has been no continuous architectural tradition. Most of the advanced Roman building methods were completely lost around the 5th century (concrete was not rediscovered until the 18th century). The Egyptians around the time of Cleopatra had no idea about the construction techniques used to build the Great Pyramid of Giza (this is why the Greeks assumed the pyramids were built by slave laborers). This is not surprising considering that Cleopatra lived closer to the first moon landing than she did to the construction of that pyramid.

[Retric]

At the smaller scale. There is a continuous tradition in home construction over that time period. Further, Japan for example had an independent tradition unaffected by Rome's rise and fall.