[bsaunder]

Almost all bad estimates are due to a lack of information. In software development, the information that tends to be missing are the particulars of the task being estimated (which code needs to be modified, in what way). This missing information is highly unique and not to generally transitive from one programing assignment to the next. Additional information about the process is usually not the problem. Things like CMM tend to focus on the process information rather than the particulars of the coding problem at hand. Which, leads many developers to think: "You're doing it wrong."

[vermasque]

Speaking from experience at a large multi-year project at a BigCo, I am quite familiar with the failures of estimation. Some information is absent because some things are simply unknown due to inexperience or inability to predict future changes or mistakes (reasonable). However, other things are lacking because there is no motivation or demand to produce them. For example, waterfall processes generally throw the software development lifecycle on a schedule for a system or subsystem: requirements, design, implementation, test. If the lifecycle stages are not divided into tasks upfront (which then should be further divided), then the complexity of each stage is basically not being factored into the estimates for each stage. This is setting the stage for failure unless it is a project that has been done repeatedly over many years in the organization.

Furthermore, continuous process improvement and monitoring is not done enough even in organizations that declare achievement of higher levels of CMMI. Our team had subsystems that overran their initial estimates by over 100%, causing nearly a year in delay. However, there were basically no penalties or major process changes despite the fact that multiple subsystems overran estimates multiple times. Process professionals, engineers, and managers are simply not aggressive in tackling these problems.

In addition, estimates tend to come from individuals. I imagine that a more collaborative team-based estimation approach would be better, factoring different levels of experience and sharing the burden of making estimations realistic. Also, recorded estimates need to be coupled with recorded justification.

[rahoulb]

agreed. we don't have a language to define the problem in the first place.

i am great at estimating - unfortunately i often go over because what i heard was being asked for is subtly different to what the client thought they were asking for