[Kenji]

We get the software we deserve. Apparently, people aren't ready to spend more money on performance, reliablility and security. When they buy software, they pay the price of a giant ball of mud (if they aren't expecting it to be free!) and they get a giant ball of mud. Simple. You call it insanity, I call it market, supply and demand. Of course, the perfectionists are unhappy about that (being one myself I can certainly empathise) but you have to realize that this kind of perfection in software costs huge amounts of money and requires the most skilled workers in the field.

[wvenable]

I've only recently started working for larger more corporate-type firms and it's absolutely amazing how much they pay for giant ball of mud software. Astronomical amounts and then on top of that there are maintenance fees, support fees, upgrade fees. It's absolutely amazing and frightening the money being spent.

So I think the problem isn't money; the real problem is there just isn't enough software! We have a software deficit. With software being a critical part of nearly every business in the world, there is huge demand but massively insufficient supply. In my industry, there are only two big software providers and a handful of smaller ones. And it's a huge industry in both money and sheer size. But they literally cannot buy anything but big balls of mud.

People are paying big money and small money for whatever software they can get. And that natural consequence is that performance, reliability, and security are hit or miss.

[Terr_]

Speaking from several years at a company pre- and post-merger, the software they want will always be scarce, because they want something customized just to themselves.

They might not get it, of course -- the money is an (often ineffective) stand-in for institutional/organizational change, AKA "throwing money at the problem."

It's simply easier to get people to spend gobs money than to change their comfortable short-term habits. (Evidence: Wasted gym memberships everywhere.)

[3pt14159]

The core problem isn't that there is a lack of software creation; there is a lack of software but the problem is that the buy is uneducated and conflates priorities. The most successful software services firms aren't those with great engineers, they are those with great sales guys.

The lack of discernment of the buyer is creating all this software waste. This is one of the reasons that MS Excel and Access run most of the world's financial data. The format is open enough that it basically imbues the user partial programming ability without having to worry about things like new line feeds and uptime.

[sethammons]

if you don't mind sharing, what industry/niche?

[wvenable]

Law.

[Animats]

"We get the software we deserve."

Painfully true. There are techniques for writing better software. They work. They take longer and cost more. They are not widely used outside aerospace.

[dgemm]

Is there a high level overview somewhere I can read? Don't even know what to google here.

[throwawayf2o]

http://spinroot.com/gerard/pdf/P10.pdf (some are more/less appropriate for non-embedded-systems)

In general, the answer typically involves formal specification and formal methods that check the code against these specifications, combined with testing and coding standards that result in analysable code.

More references:

https://www.cs.umd.edu/~mvz/cmsc630/clarke96formal.pdf

http://research.microsoft.com/en-us/um/people/lamport/tla/fo...

[tormeh]

You might want to look into coding standards for C and special languages like Ada (like C, but less writeable, more readable with strong types) and Esterel (deterministic multithread scheduling). Seriously, Esterel is probably the coolest thing you'll read about this week.

There's also various specification languages for multithreaded behaviour, which allows you to analyse your programs behaviour using software tools, for example SPIN[0].

0: http://en.wikipedia.org/wiki/SPIN_model_checker

[pjmlp]

Search for high integrity software.

For example, with MISRA it may be C, but feels like Ada.

http://www.misra.org.uk/

Or Spark similarly for Ada

http://www.spark-2014.org/

[rapind]

If we're all creating giant balls of mud because the demand is for giant balls of mud done quickly, then we never learn how to craft beautiful software.

It's unfortunate, and it's sad to say I'm often part of it at some level (but I fight it dammit!).

[auxym]

Also in medical devices. IIRC FDA has standards for software writing/testing/certification.

[mcmancini]

FDA requirements mostly target the SDLC: risk analysis, change control, documentation, v&v, &c. Companies are afforded a surprising amount of flexibility in implementation. Basically, you must have a documented process that you follow, but you're left to your own devices in creating the process. Deviations from voluntary industry standards (e.g., TIR 45) is permissible since they're not specifically required. The DoD, aerospace, and automotive industries have in comparison far more detailed and strict regulatory requirements.

[FLUX-YOU]

I'm not sure how much faith I'd put in that.

http://www.wired.com/2014/04/hospital-equipment-vulnerable/

http://en.wikipedia.org/wiki/Barnaby_Jack#Insulin_pumps

[roma1n]

Yup. How a DO-178-like integrity level is not mandatory for medical devices is troubling.

[flatfilefan]

I guess the other thing is instability of the requirements. Something like grep or uniq are almost perfect because the use case is stable.

[krylon]

When I was still in training, I had the pleasure of participating on a software project to build a "web shop" for internal use at a large IT company. It was a pleasure insofar I was only a trainee and did not have to deal with the politics and all that. One day, a week before the first part of the web shop was scheduled to go live, the programmer I was working for came back from a meeting. Usually, he was pretty laid back person, very friendly and pleasure to work with/for. But when he came back from that meeting, he was mad, like, "Hulk SMASH" mad.

I asked him, as delicately as I could, what had happened.

This project we were working on, he told me, had been going on for about half a year, and during that time there had been a meeting every week, where everybody involved in the project had an opportunity to get together and review and discuss the progress of the project.

So there was this guy, who had only attended the very first of these meetings, without saying a single word. And the week before the thing is supposed to go live, he shows up again, with a long list of changes he wants to make to the application.

Needless to say, that programmer, and several other people in the room, told that guy how he had six freaking months to tell the programmers about his requirements, and the he could not possibly expect them to make all these changes a week before the application went live.

So that guy goes to the highest-ranking manager involved with the project, manages to pull some political strings, so that manager goes to this programmer's boss, that boss goes to the programmer and tells him, "I know how much it sucks, but you have to do this. I share your pain, but I, too, am powerless to refuse this request."

When building material things, it seems, houses or ships or airplanes or railraods or whatever, people do realize that you cannot tell, e.g. the construction company that you actually want a house with a circular outline rather than the rectangular one the company has been building for the last, what?, three months. Managers at car companies, I guess, do not storm into the engineers' offices a week before production of a new car starts to tell them the vehicle not only needs to be small and fuel efficient, but also needs to be able to work in antarctic climate and run on carbon dioxide instead of gasoline.

With other products - at least, that is the impression I get - people have an intuitive or explicit understanding of the limitations of the things they want to have built, and they also seem to basically understand that you cannot make drastic changes at the last minute. With software, which does exist in the tangible way that, say, a car our house does, people seem to have a much harder time understanding those limitations. I am not sure this is the entire problem with software development, but it is a big part.

[dasil003]

Thankfully I'm at the point in my career where, if such a thing happens, I can respond calmly and without anger that this will reset the clock on development, and that doing so is a management decision that does not reflect poorly on me or my team. If they want to do that, I can do that, but I won't take the blame for that failure.

I recognize some organizations will not accept that answer and will try to make me take the blame. Such organizations will tend to have a very poor tech department.

[lordnacho]

The thing with programming and abstract stuff is it's very hard to know what makes the problem much harder. And you definitely won't have intuition for it if you haven't programmed.

For instance, it's easy to prove that there's an infinite number of prime numbers. There's no biggest prime, because if there were (hand-wave proof coming) you could multiple them all together, add one, and have another prime number. Easy right?

Now tell me if there's also an infinite number of twin primes, ie pairs of prime numbers separated by two.

Here's a good xkcd: http://xkcd.com/1425/

[wyager]

>but you have to realize that this kind of perfection in software costs huge amounts of money

Maybe not as much as people think. There are machine-assisted mechanisms for building mathematically perfect software. See Curry-Howard oriented languages like Coq and Agda. It's certainly harder to write perfectly, provably correct programs, but maybe not as hard as most people think.

There is also a lot of middle ground in the form of relatively powerful (but not quite Agda-powerful) type systems.

[vezzy-fnord]

Mathematically perfect provided the proposition you're verifying is actually relevant and consistent with the one you're applying in the real world. Given the finicky nature and constraints of the environments that most software runs on, this could prove difficult. Moreover, the proofs must be maintained along with the software.

As for type systems, mission-critical software has generally made good practice with them (Ada, proof checkers, etc.) More so than consumer and enterprise circles. Yet a ton of catastrophic bugs have been the result of errors outside the scope of type checking, and as a counterpoint, our telephony is pretty robust with dynamically typed Erlang switches. It seems that mechanisms for building self-healing and concurrent systems are often put aside and equivocated solely with strong type systems. From what I recall, Erlang's signature feature of hot code loading actually conflicts with static typing.

[Kenji]

I am familiar with some code-proving tools and functional programming. Verifying such code is tedious and will increase time, cost and skill required for the project substantially. People just aren't ready to pay the price for that. Yet.

[tormeh]

The thing about typechecking, though, is that you're essentially specifying your program's behavior twice: Once very detailed in the actual code and once at usually some lesser detail in the types. All the compiler does is checking that these two specifications are consistent with each other. Your types can have bugs too, in which case nothing will help you. What frustrates me about detailed type systems is that as detail increases, difficulty of writing the types will increase and type bugs will become more common. Now, assuming the bugs in your code and types are statistically independent that would still save you a lot of bugs, but I suspect they are not.

[Dewie]

> The thing about typechecking, though, is that you're essentially specifying your program's behavior twice: Once very detailed in the actual code and once at usually some lesser detail in the types.

How so? The implementation is not the specification; but the type is the specification. Like people like to say, the type is a Theorem, and the implementation is is the Proof of the Theorem. (In a very real sense.) And don't you need both?

Hopefully there can be some type inference at a certain scope, to avoid cluttering the code with a lot of 'obvious' type declarations. I forget exactly how dependently typed languages work in that regard right now (there certainly can't be full type inference).

> All the compiler does is checking that these two specifications are consistent with each other.

"All". That's already better than informal mathematical proofs.

> Your types can have bugs too, in which case nothing will help you.

Yes, just like any other kind of alternative specification there is. Short of mind reading, there is no getting around actually describing what we want. But where there is a distinction to be made is in what kind of specification is easy and declarative enough to use as to give the least likelihood of introducing bugs in the spec itself.

> What frustrates me about detailed type systems is that as detail increases, difficulty of writing the types will increase and type bugs will become more common.

I guess if we assume that we have collapsed/unified the type and term(/value) level, we can use any old Good Software Engineering Practice when it comes to keeping the types tidy. Like using type functions to encapsulate some of the detail: maybe have a `sorted(list)` function instead of having to write it out each time we need it.

This is just a guess though; I don't know how the dependent programmers do it. Well most of what I'm writing here is guessing, on some level.

(And judging by the pride that some people show when they proclaim that "half of our code base consists of just tests", well... I certainly think that types can be more succinct than that!)

> Now, assuming the bugs in your code and types are statistically independent that would still save you a lot of bugs, but I suspect they are not.

I guess if we go with the previous assumptions of a unified value/term level, then the statistical chances are the same in that types are just ordinary values, instead of the types belonging to its own language. ;) Then we just have to make sure that the amount of types is substantially smaller than the amount of "regular ol' code".

Typer er fremtiden. Håper jeg (kanskje).

[Dewie]

The problem with advanced type systems is that, while the theory and implementations and the research might be mature, it will take god-knows how many years of research oriented on usability in order to get something approachable and understandable. If people are even researching usability like that in a systematic manner (I have my doubts whether there are PL researchers that are working on that kind of thing, or if the community has just thrown their hands up and said that it isn't doable).

[Varcht]

So true, pretty much these thoughts went through my head every time my last boss lectured me to get the team to produce faster while reiterating his bug free goal.