[davestewart]

The article description is "A deep dive on why projects always overrun and a framework to improve future estimation"

This was a personal investigation into my faulty estimation skills, off the back of a small project which became a medium project, which became a large project with no shortage of surprises, overruns and pain.

The blog post was born from an honest and thorough postmortem where it turns out most of the work was simply not expected – and so wasn't accounted for.

I then went a stage further and attempted to outline more general reasons for this and to visualise how might look in terns of time and effort. It's not meant to be scientific, but is certainly an interesting way to look at things.

Anyway, I hope someone finds it useful.

[rgbgraph]

Golly gee, someone that realizes software engineering isn't entirely just sitting down and pumping out code? Why aren't you in management? (don't tell me... it's because you're honest with yourself)

I do wish this becomes more popular (and the graphic does help for those without imagination ;).

All venting aside, if you're familiar with a system, its development and deployment environment, and all the quirks of the tools you're using, you can easily "feel" around for how much "dark matter" work you're going to have to do -- and your estimates become a lot more tight, and close to reality. That is, once you throw away -- as you said -- "happy path" optimism and get down to business. An extra padding of cynicism will make sure you never miss a deadline; and if it turns out you were too cynical, you can always under-promise and over-deliver and wow everyone.

But padding is necessary, and I wish it didn't have this air of dishonesty associated with it. Realistically, shit happens. Stakeholders can huff and haw because it gets in the way of their aggressive plans, but it's not dishonest to say it's going to take another 3 months, because you're asking me to dig you a hole with a plastic sand-castle shovel. Yes, I can actually pound Red Bulls and wear myself down to the bone digging that hole as fast as humanly possible -- but I'm not going to.

If that's not good enough, the next best is to ship in phases (what little "A" agile tries to do, but fails in practice): a group of features will be guaranteed to ship at some date. At that point, we'll reassess and estimate the next batch -- until the project is done. We can even do soft estimates on all the batches, but using ranges (e.g. 3-6 months, rather than something concrete -- because they'll become de-facto deadlines), to give a rough "total project estimate." I like the month-by-month "sprint" model more than the 2 week model. Analysis and planning isn't free. It costs time, effort, and mental resources. Doing it every 2 weeks is absolutely ridiculous. Monthly strikes a nice balance between giving you enough time to actually do the "The work" and give stakeholders a feeling that they're "on top of things."

[fatnoah]

> Golly gee, someone that realizes software engineering isn't entirely just sitting down and pumping out code? Why aren't you in management? (don't tell me... it's because you're honest with yourself)

My ironic situation right now is that I AM management in my company, and I'm spending a lot of time shifting the culture of our Engineers away from "just pumping out code" to shipping and supporting products. I'm literally being asked questions about why we need logging when we can just use a debugger, and I'm fighting the "that'll only take 1/2 day" (which it never does!) estimates from the team.

It feels like bizarro world to me. I had a pretty long IC career before moving into management, and felt like I was fighting the exact same battles against management,

[spockz]

In find that even though I’m typically pretty spot on in the amount of actual design and coding time a feature requires, the actual wall clock time can be way off. Ad hoc meetings, days off for holidays/corporate events, waiting on external feedback, etc. are all very hard to account for.

[TeMPOraL]

Something I realized while skimming this thread, and moments before opening the article itself - I'm definitely underestimating things by factor of 2x, because I keep forgetting about the denominator - that my one work day isn't really 8 people-hours of project work!

In reality, it's closer to 3-4 people-hours on average, after I account for time consumed by team meetings, corporate paperwork, 1-on-1s, code reviews, lunch break, help requests from teammates, IT/devops doing maintenance on infrastructure, requests to opinionate on / get involved with some discussions about new projects or with new customers... - a lot of work that's mostly necessary, but isn't relevant to the particular thing I'm focusing on (or estimating).

So on top of the excellent framework from the article, I'm going to keep reminding myself that, for the purpose of estimation and project work, I'm working 3-hour days, not 8-hour days (and that's before we factor in any fuzzy human stuff like kids getting sick, becoming burned out, etc.).

[zmgsabst]

I have the same 3-4 hour estimate, but I think this stood out to me because of working a labor job as my first:

- 8.5 hours on site

- 1 hour of breaks + lunch

- 30 minutes of getting into/out of lunch (tidying up + restarting)

- 30 minutes of gathering tools/supplies in the morning

- 30 minutes of putting stuff away at close

- 30 minutes walking around/getting stuff you didn’t expect to need

- 30 minutes of talking to boss/coworkers about their problem or yours

So you’re talking 5 hours of “real” labor in an 8.5 hour day — and that’s on days you have a consistent project. Days where you have a bunch of small tasks all over the apartment complex might be 3 hours of “swinging hammers”.

That’s normal — and I think people tend to forget process when estimating things.

[Swizec]

> and a framework to improve future estimation

The best framework I've heard of and used in practice is to simply ask one question: "How long have similar projects taken in the past?"

Chances are this one is no exception. It will take about the same time as similar past projects.

[bazmattaz]

Thanks Dave, as a PM this is super useful to me to help me emphasise with my team.

I love your illustration of the work involved to get a project live. I think I might bookmark this for when I encounter that pushy stakeholder that wants a fixed estimate

[davestewart]

You're very welcome!