[shriver]

While I agree with you that software projects aren't unique in having difficulty trying to estimate things, I think that emotionally it's different. If you're digging a tunnel and hit a seam of unobtanium you can go to your boss and point at the seam and say 'Look at this big rock - this big rock is very hard, we had no way of knowing it was here but now we know we have a problem". That is very different from going to your boss and saying "You know I was working on X, well X is harder than I thought, it's going to take longer". Well - is X harder? Or are you just slacking off? Or maybe you just took a stupid approach?

It's much more difficult to actually confirm that the cause of the delay is an external factor, rather than a reflection of the person doing the work since most likely the only person with a real understanding of exactly why something is difficult is the person doing the work. So there's a stigma attached that isn't present in other professions.

[jdmichal]

You will never know what you don't know. That doesn't change. The question is, what do you do about it? That tunneling project probably did account encountering different materials, but maybe not for unobtainium. If the area is known for unobtainium, that's obviously a research miss. If this is the first seam ever encountered within 500 miles, that's straight up unexpected findings. Most situations usually lie somewhere in between.

You can always do more research, but research is at the cost of getting things done. So you balance that, with the understanding that sometimes you will go into projects with too little research. And no matter how much research you do, you will occasionally get genuine unexpected findings while doing the work that cause you to stop and go back into research mode.

EDIT: I mean, just watch those DIY shows some. Almost every episode, there is something "unexpected". Foundation work, roof work, load bearing walls, pipes and vents in walls, electrical or plumbing needs updating, etc. And it rarely ever actually effects their timeline, because they've done it enough to model that something unexpected is probably going to happen.

[52-6F-62]

I wholeheartedly agree. I think it’s however more of an unacceptable problem in non technical industries who hire technical workers.

For instance, risk assessment isn’t all that helpful if your deadlines are bumped up by weeks or months and the client or your coworkers or bosses have no concept of why the project “can’t just be done immediately” when you’ve planned for added weeks. Even worse when they don’t understand why they can’t suddemly throw new variables at you after all estimates have been completed like new, potentially breaking features. Last minute scope creep is a real problem in some non-technical fields because the whole software solution is often some “nerd magic” and there’s no desire to understand the problem or solution except where to click and can they have it all by the end of the week. Theoretically maybe it could be done, but there goes all window provided for unexpecteds.

Admittedly, this is probably more of an organizational flaw than directly associated with the subject at hand, but a very real tangent.

[marcus_holmes]

I usually try to explain this using a sporting analogy, because the people who treat this as a "nerd problem" usually understand and relate to sporting analogies.

Why can't you hit a hole-in-one on every shot on a golf course? You know exactly where the ball is, where the hole is, how far it is, which way the wind is blowing. All you have to do is hit the ball so it goes in the hole. It's definitely possible, there have been hole-in-ones on every green, so why can't you do that for this shot? Are you an incompetent golf player? Are there super-competent golf players who can do this on every shot? Why not? This does not seem difficult to me (but then I've never hit a golf ball in my life). Can you explain why you can't do this, but other people have done it before?

[52-6F-62]

That's pretty good. Hope you don't mind if I end up using that or a variation in the future.

[repolfx]

I'm not sure argument by reality TV is very effective. Obviously everyone tries to budget time for the unexpected. That doesn't mean it's equally difficult in all professions.

[gregoriol]

You are referring to non-engineering managers, which will also happen in other industries, this depends more on the size of the company I'd say.

Also, if you are really the only one to understand (if you are the only tech in a startup for example), then part of your job is actually to explain.

Problems hit by developers are imho the same kinds that you describe with rocks: something unexpected has happened, we didn't/couldn't see it coming because of this and this, we need to do/change this and this, ... or maybe the risk was already considered in the estimate.

One big difference I was thinking about is that in the software industry nobody dies if something goes wrong. In building a tunnel, or shipping or many other industries, if you don't consider the risks well enough and ahead, someone might die.

[ianwalter]

They're not even necessarily referring to non-engineering managers. It's often difficult to explain to engineering managers or other engineers why something won't work or requires more work/research if they themselves have not done the work. The engineering manager isn't even necessarily that familiar with the technologies being used or experienced in the specific problem being solved.

When digging a tunnel, there is a finite number of kinds of rock that you can hit. It's well within the realm of possibility that an experienced engineer is familiar with any of the rock they will encounter, even if they didn't expect to hit them when digging. This is generally not the case with software. Even as an experienced engineer you continually hit issues that you've never encountered before and don't know how to solve without spending a significant amount of time doing research or trying out different solutions.

The upper bounds of complexity are higher in software than construction. It's not narcissism, it's just the nature of that kind of work.

[agentultra]

> The upper bounds of complexity are higher in software than construction. It's not narcissism, it's just the nature of that kind of work.

It's a wonder that it works so well in fact.

I think software is probably the most complex and difficult engineering humans undertake. If you think of all the ways that a program can crash, compute incorrect results, and generally go wrong -- it's amazing that it even works at all let alone that in practice it works so well.

And we don't even have a widely accepted body of knowledge, capital-P professional obligations, liability, etc. We constantly hire people with no formal training, give them no formal training, and make no expectations of their expertise. We hire them for lots of money and put them in front of important projects and we promote them up. It's almost unreal.

Talk to a civil engineer or a doctor about what it took them to enter their profession and the amount of work they have to put in to maintain their license. And they get paid less than most software engineers.

And yet it's still somewhat unbelievable. Digging tunnels requires a lot of expertise and domain knowledge but as others have mentioned: there are only so many rocks out there to encounter. Once you have a thorough survey there are scant few unexpected problems you might run into.

But computing? Well... unless there's a battle-tested library that solves the exact problem you have then you're basically solving the problem from scratch. And one mistake is all it takes. You can't just add an extra I-beam or use a stronger steel... there's no fudge-factor in discrete systems. The whole enterprise of computing is managing complexity -- we have whole fields dedicated to the study of complexity with plenty of unsolved problems!

I believe it is just the nature of the work and why it remains so infuriatingly difficult to predict outcomes.

My pet response when someone asks why estimating software is hard: you've lost your keys in your house. give me an estimate on how long you think it will take to find them. I know you're going to think that if the keys are in the usual place it will take less than a minute. But if they're not... If you check all the places you might've been in the last few hours, maybe 5 minutes. But if they're not there then you'll have to think about how to search your whole house which could take hours. And what if you still don't find them?

And on and on.

[chli]

Nobody will die if your webpage doesn't render but software is used in safety critical environments too.

[XorNot]

Safety critical environments though are just fundamentally different when software is developed. Like a reality of them is that the assumption is a development time of years if the needed platform doesn't exist, and if it does then the changes need to be trivial to the point of being done by the time you propose doing them since all your time will be spent analyzing and studying the impacts.

[gregoriol]

I meant those who make the project don't die in software

[alxlaz]

> since most likely the only person with a real understanding of exactly why something is difficult is the person doing the work.

This is often the case in other engineering projects as well.

But there's often less of an incentive to "confirm" that this is the case because if you (claim to) have hired the best people, to have given them the best possible training and working conditions, and every support in doing their job, it rarely makes any sense to go to them and figure out if they're really late because the problem is hard, or because they're just slacking off.

The investigation itself is rarely meaningful. Even if they are slacking off, what exactly do you expect they'll say? Yeah boss, we're just slacking off?

Managers who second-guess perfectly competent teams this way are usually shown the door (if they ever made it to managing something).

[nautilus12]

Exactly this