[saltyoutburst]

We haven't given up, there is simply not enough financial incentive to make the software any better. See 'We Could Write Nearly Perfect Software but We Choose Not to' https://blog.inf.ed.ac.uk/sapm/2014/03/14/we-could-write-nea... "The simple truth is that bug–free on-time software is just more expensive than we (or our clients) are prepared to pay."

[kls]

The sad part is the authors example is easily avoidable and does not cost more to avoid. The fact that a developer was making manual changes in a production environment points to bad process and lack of knowledge of a better way. In every environment I have been in since 2000 production is locked down it is modified by scripts that are tested in several environments before the script even gets to production. With virtualization and continuous integration the cost of doing it correctly is marginal and the savings in avoiding crisis and recovery is substantial. There in lies the core issue, it seems more and more developers are loosing focus of the process of managing change across environments and are resorting to manual processes that are not reproducibly. Simple point is, if you are making manual changes in any environment other than a local dev environment you are doing it wrong.

Source control should be your single source of change, and check-ins should cause a chain of events via continuous integration and issue tracking. A checkin should cause a build to kick off if it is successful it should deploy to an integration environment where tests should smoke test it. If it smoke tests it continuous integration should up merge to a test branch, deploy to test and then update the ticket system to update all associated tickets to ready for test. When the tester verify the tickets automation should again up merge to release and automation should release to production on a release schedule. This is a solved problem.

[jack9]

> The sad part is the authors example is easily avoidable and does not cost more to avoid.

Except for all the costs of maintenance. It always costs more, generally in the way of time.

> With virtualization and continuous integration the cost of doing it correctly is marginal

Until you have to debug while the integration fails.

> This is a solved problem.

Even with unlimited funds, you get a cost of time and communication. There are tickets and tickets and a queue and a release schedule. The optimal path is not as simplistic as you would fantasize.

[d2p]

> The fact that a developer was making manual changes in a production environment points to bad process and lack of knowledge of a better way.

I'm guessing this is relating to my SQL mishap in the opening lines? You're making assumptions that I was doing something manually in production here, and not that I was executing a script that was buggy or against the wrong machine.

Sure, we could build safeguards against this, but it wasn't something we really considered would happen. Needless to say, we've learned from that one!

[x5n1]

Hot patching is a standard practice in production. It's sometimes necessary.

[ereyes01]

Great link and article. I've always thought the NASA story is truly fascinating. Knowing that your bug could kill the person sitting across the conference table certainly changes the attitude towards being careful with software :-)

As others have said, software is a young field, and in many ways we're still pushing and exploring the boundaries of what's possible with software. All of us are still very much "early adopters" in this nascent technology experiment / revolution. The nature of demand and untapped potential for software systems creates a financial incentive for high-stakes rapid experimentation at the expense of sloppiness (in attitude and in software quality) we've seen for as long as there's been software.

I do think investing a little bit more time up front and using TDD combined with good leadership / experience is ultimately healthy for your product and team, if you can manage it. However, we shouldn't forget that being "cowboys" got us to where we are today, and continue to break open exciting new opportunities for software.

[catnaroek]

This is misleading. Most of us couldn't possibly write significantly better software, even without time and money constraints. Money can buy existing talent, but it can't create talent where there is none.

[lmm]

Nonsense. We're in the software industry. We can use tools, and if the tools aren't good enough we can make our own. Writing correct software isn't a matter of talent, it's a matter of process.

[catnaroek]

Writing correct software isn't just a matter of having the right tools and processes, even if those can help. It's primarily a know-how issue. No amount of process can fix using the wrong algorithms or coming up with the wrong designs.

[lmm]

Suboptimal algorithms may lead to poor performance but you can enforce correctness. If you let the requirements drive the design rather than trying to design up-front (i.e. good process) you end up with good design. Asking the right questions to gather the correct requirements is arguably in-score or not, but again it's primarily a process issue.

[sievebrain]

You can't always enforce correctness. Consider the case of a typical web dev shop that hires a developer who thinks escaping is a good solution to SQL injection. It might get snuck past code review if you're unlucky.

There are static analysis tools that can identify such problems ... sometimes ... maybe ... depending on the language and frameworks in use. But to deploy such tools you have to know about them to begin with. And most devs can't simply produce such a tool if there isn't one on the market because that's not what they're being paid to do.

Unfortunately, knowhow can't be automated away entirely just yet.

[lmm]

> Consider the case of a typical web dev shop that hires a developer who thinks escaping is a good solution to SQL injection. It might get snuck past code review if you're unlucky. > There are static analysis tools that can identify such problems ... sometimes ... maybe ... depending on the language and frameworks in use.

Analysis tools are the wrong approach - that code should be simply impossible if you're typing things correctly, and inadequately typed code should be very obvious in code review.

> And most devs can't simply produce such a tool if there isn't one on the market because that's not what they're being paid to do.

Most devs could write such a thing if the company told them to. The reason "that's not what they're being paid to do" comes down to process.

[catnaroek]

> If you let the requirements drive the design rather than trying to design up-front

I don't understand this dichotomy. What kind of engineer designs anything for a third party without gathering their requirements?

[lmm]

I meant more that you need to let the things you encounter during implementation drive the design. This is particularly relevant when requirements change (as they often do), but even if you can nail down the spec perfectly up-front, trying to design at that stage is still a mistake.

[srtjstjsj]

wrong algorithms and wrong designs are not the cause of bugs and catastrophic failures. Simple unit mis-conversions and boundary cases are.

[bdamm]

Is there some reason to believe that with enough money software could be bug-free? Developers are often guilty of just refactoring for no particularly good reason other than their personal sense of aesthetic, which can lead to endless (and destructive) refactoring. More money also makes people design lazy, so that instead of finding clever ways to reduce effort and improve uniformity, you just have everyone make their own forms.

[diafygi]

There is. See the space shuttle software.

http://www.fastcompany.com/28121/they-write-right-stuff

[nightski]

It isn't a practical solution for the majority of software out there. I also think that while their method works, it is pretty archaic. Looking to the future I hope we can find a way to achieve the level of results they do, without the ridiculous effort and cost.

[adrianN]

I work on safety critical systems. At least 80% of the work for building such a system is not critical for the actual quality of the product, but is necessary to satisfy compliance rules. Documentation of the development process is really extensive.

[pron]

> Looking to the future I hope we can find a way to achieve the level of results they do, without the ridiculous effort and cost.

That is unlikely. For computational complexity theory reasons, writing correct software is extremely computationally expensive (regardless of whether or not the language is Turing complete). In fact, it is "the hardest problem in computer science", in the sense that any problem with bounded complexity can be efficiently reduced to software verification. Even verifying finite-state-machines (the simplest computational model) is PSPACE-hard.

What we can do is find many ad hoc ways, each helping to some extent with some kinds of correctness properties.

[nightski]

Most software problems I encounter in the real world ARE of bounded complexity, and they can be reduced to software verification. But this is still a very tedious process and there are significant gains to be had by making this more accessible without having to solve the hardest problem in computer science.

[pron]

Certainly. It's just that there cannot be one (or a few) advances that would make writing correct programs generally easy. We can concentrate on domains (in the safety-critical embedded world this is almost a solved problem for some kinds of applications thanks to the invention of synchronous languages in the '80s, that make formal verification relatively efficient). We can also address some "internal" properties, like memory safety and transactions, that on the whole might make writing correct programs easier (though not provably correct).

[FigBug]

And yet it still couldn't handle flying over new years: https://www.newscientist.com/article/dn10459-y2k-like-fears-...

[gsnedders]

It probably could and in testing it could, they just decided not to find out whether it would really work properly when seven people's lives depended on it working. If you can practically avoid an edge-case you haven't ever relied on in decades before when people's lives depend on it, not risking it doesn't seem that crazy.

[chris_wot]

Actually, they could but given the fact that the shuttle is not much different to strapping equipment and people to a gigantic bomb they seem to have decided that even though exhaustive testing found no issues a small change in the schedule was less risky.

[protomyth]

> Is there some reason to believe that with enough money software could be bug-free?

NASA with the Space Shuttle software came pretty close. http://history.nasa.gov/computers/Ch4-5.html

[kruczek]

In case of NASA, it's not only because of money though. It's because of a very rigorous process of designing and testing their software, sufficient time to follow this process, as well as writing a single product for a single customer. Of course money enables such process, but alone it is not sufficient.

[marcosdumay]

Enough money and the goal of making software bug free can make it bug free.

But enough money and the goal of making it infinitely flexible, having all the possible features, or the most perfect UX that a team can not decide on what it is will only make the software more bloated than it's reasonable, and fill it with bugs.

[panic]

I think "bug-free" software is a distraction. It's fine to make the occasional mistake. The real problem here is that nobody seems to take these mistakes seriously. Everyone seems to expect software to be occasionally broken, including the people writing it.

[catnaroek]

Aren't your second and last sentences in conflict? How could we not expect software to be "occasionally" broken, if it's "fine" to make the "occasional" mistake?

Of course, making mistakes is only human, but mistakes should never get past the build process (compilation, automated testing). Any mistake that does reflects an error in your design (not making your code amenable to verification) and/or your process (not capturing requirements in tests).

[panic]

Yeah, using "occasionally" in both places drew a parallel that I didn't intend. We can expect people to make mistakes without accepting broken software. As you say, there are processes that can reduce the impact of mistakes on the final product. I think a culture of taking issues seriously is also important. Small mistakes are easy to make, but they're easy to fix if the organization has a good way for them to be reported and allows people to spend time fixing them.

[pmarreck]

> that with enough money software could be bug-free?

According to empirical studies, practicing TDD leads to a 90% (yes, that number is correct) post-ship bug reduction at an upfront cost of 15-35% more development time.

Which seems like a pretty good tradeoff, to me. (And I've experienced it firsthand.)

[DomreiRoam]

Do you have any links on this studies?

[pmarreck]

Here is the infamous Nagappan TDD paper from Microsoft Research

http://research.microsoft.com/en-us/groups/ese/nagappan_tdd....

Also discussed in this infoq article

http://www.infoq.com/news/2009/03/TDD-Improves-Quality

Here's someone's thesis paper on it, with data:

http://www.nomachetejuggling.com/files/tdd_thesis.pdf

There's also interesting discussions like this

http://programmers.stackexchange.com/questions/206355/the-re...

"it is my experience that the value proposition for TDD grows exponentially as the time and resources involved in a project grows linearly."

This has also been my own experience.

This person's comment also is interesting and has data

http://programmers.stackexchange.com/a/210756

Some of the above is empirical (read: scientific, no-bullshit) data.

My opinion on it has grown to the point that I think TDD should literally be inseparable from programming, and the two together should simply be called, "programming." Lacking any unit tests whatsoever (TDD or otherwise) should be called, "taking stupid, extremely hazardous risks to save a little time, like reading your phone while driving"

I've not found a programming task in at least 5 years that wasn't waaaay better-written when done via TDD. Some things, like IO, can be a challenge to unit-test, but that's why we have great ideas to solve that like Gary Bernhardt's awesome "Boundaries" talk https://www.youtube.com/watch?v=yTkzNHF6rMs

[lmm]

> Is there some reason to believe that with enough money software could be bug-free?

Yes. It's pretty easy to write provably correct software, just expensive.

> Developers are often guilty of just refactoring for no particularly good reason other than their personal sense of aesthetic, which can lead to endless (and destructive) refactoring

Not my experience, but even if so, if you require your software to be proven correct then it won't matter - any refactor that breaks it simply won't be accepted.

> More money also makes people design lazy, so that instead of finding clever ways to reduce effort and improve uniformity, you just have everyone make their own forms.

Um what? What does this have to do with anything?

[majewsky]

Once you have provably correct software, the much harder problem is provably correct hardware. :)

[hyc_symas]

Good point. But choose the wrong hardware and you can never have provably correct software. E.g., the Intel x86 ISA makes all x86 processors impossible to prove correct. (there are 2^120 possible instruction encodings - the universe will die heat death before anyone could verify all of them in an implementation http://www.emulators.com/docs/nx06_rmw.htm )

[lmm]

Shrug. We have provably validated diodes AIUI. If you can get provably correct basic components you can build up complex components using the same techniques as in software.

[forgotpwtomain]

> Developers are often guilty of just refactoring for no particularly good reason other than their personal sense of aesthetic, which can lead to endless (and destructive) refactoring.

There are lots of good reasons to refactor - if indeed frivolous refactoring is more prevalent I would think this kind of opinion should be supported by a source/reference.

[kartickv]

Economics is only a part of it. Often, poorly built software increases costs for the company that built the software, like when customers call the helpline to get an agent to do what they could have done on the website. Or, when I call the helpline and the agent asks me to call again because their software isn't working today.

As another example, when I was logged in to my mutual fund web site, I found a button that says Get Statement. Since I wanted to download one, I clicked it, and it said, "Your statement will arrive in the post soon." They sent a paper statement without confirming me what I wanted. When it arrived, I threw it away, of course.

In many cases, poorly built software increases costs for the company as well, so financial incentives are only part of the story. Incompetence is another part, maybe a bigger one.

[spuz]

Considering the upfront cost has just as much to do with economics as considering the overall cost of high quality software. Economics is a description of human behaviour and all the irrationality and incompetence that comprises it so decisions of short-term gain vs long-term gain are very much included in the study of software from an economic point of view.

[kordless]

I believe the problem lies with the "typical" business models for software companies.

[TheCoelacanth]

The problem lies with the amount of financial resources available to software users. Users can afford buggy software. They can't afford non-buggy software except for a very small number of uses.

[d2p]

Often the buggy software turns out more expensive. It can be a false economy not putting that extra time in to ship stable software.

If you ship something that loses data and you have to start restoring backups and patching data, it probably would've been cheaper to do a little extra testing.

Like everything; it's a tradeoff. I think we're universally bad at playing the game!

[asimuvPR]

Curious about your point. How would this apply to free open source software? What us the equivalent of affordability in it?

[DougWebb]

The high cost of bug-free FOSS is the time the developers can volunteer to develop it, which takes away from time spent on additional features. The developer's time is a limited resource that has to be spent where it is best utilized, and making the software bug-free might not be the right place.

[asimuvPR]

Yes, good point. In both cases the limit is time. How do you think one should decide on what qualifies as best utilized? What would count as additional features? Does documentation count as a feature in software?

[kordless]

If there is anything sacred about software, it's the documentation.