[bart3r]

I'm surprised there is no reference to time estimation. An important part of their role is estimating how long a task will take to complete, and I've found many people, even engineers with a lot of experience, are terrible at this.

[Geminidog]

People in general are terrible at predicting the future... I don't think being clairvoyant is a quality that should be expected out of an engineer or anybody.

This whole time estimation thing is akin to predicting when the next hurricane or earthquake will occur. The main problem is business people don't understand that so they place this unrealistic burden on engineers.

A manager or business guy who needs constant and very accurate time predictions is a sign of a bad manager that is overly reliant on engineers and lacks understanding of software. A good manager should have the technical knowledge to make a technical guesstimate himself (that will also likely be wrong) and have the foresight to be able to manage delays and allow for buffer time.

A great team of people creating a product consists of both great Technical product managers and great software engineers. A rockstar software Engineer alone may not have the ability to manage the politics of unrealistic expectations.

[username90]

> This whole time estimation thing is akin to predicting when the next hurricane or earthquake will occur.

This myth needs to die. Can you predict if an item will take closer to a month or a decade? If true then it is far easier to predict than hurricanes or earthquakes. You might not make predictions as accurate as management wants all the time, but most can predict how long things will take within a factor of 3x or so and it will be within that margin most of the time, a person who could do that for hurricanes or earthquakes would be the greatest genius in history.

And yes as you get more skilled your predictions will become more accurate. Hence accurate predictions being a sign of skill.

[Geminidog]

>This myth needs to die.

Let me spell it out in an example. Sports. Horse racing or basketball. You have a team of highly skilled players with a bunch of information quantized, including height, weight, score statistics, rebound statistics, biography... etc. And these guys are in a game with a very very controlled set of rules under exactly the same time pressure and everyone still fails to predict the outcome.

In software you have a product. The product is usually not concretely defined and you have a complex code base and you can never be 100% sure exactly how the new product will integrate with that code base... you're also not 100% sure how the code will be put together to define the product. Additionally are you 100% familiar with the stack? Do you know every possible primitive of psql or ruby or python or C++ that you could be using to create your project because I pretty much guarantee you every basketball player more or less knows every possible move and rule of a basketball game.

You're also working with a team that includes people that you have much less information on than normal. You worked with a guy for what at most two years does that give you accurate statistical information to the degree of say a basketball player? Also there's bound to be people you're less familiar with working on the project as well. Are you interacting with other teams as well? Does the outcome of your project hinge on the completion of a feature by an entire team outside of your own?

People can be experts on horse races or sports. Even then they can't predict things accurately. If you were to start making bets on software development dates of completion. You will also massively fail because not only are there more variables in a software project... but you have much less information.

Chaos is a phenomenon that happens to systems we have close to perfect information for. We find that if we have the perfect information of all the particles in a weather system except for say one particle. We find that information about that missing particle will make our mathematical calculation wildly inaccurate.

For software we don't even have anything close to perfect information in a system with multitudes of variables. Chaos will throw any prediction off.

>but most can predict how long things will take within a factor of 3x or so and it will be within that margin most of the time, a person who could do that for hurricanes or earthquakes would be the greatest genius in history.

3x of what. 3x can be big or small depending on x. So if I predict a project will take one year I can be off by 3 years under your logic. If I predict a month, than I can be off by 3 months. If I predict a week, 3 weeks. 3x is pretty horrible if you ask me, it's easy to make guesses within these parameters.

I predict that both a hurricane and an earthquake will happen in a century. I'll only be off by 3x or 3 centuries. Actually I can do better than that. I'm 100% sure multiple earthquakes and multiple hurricanes will happen in the next century and I am 100% sure that I will by 0x off let alone 3x.... Look I'm the greatest genius in history.

3x is not a reasonable margin of error.

[Benjammer]

Thanks for this, I'm going to use the sports analogy in the (near) future the next time I discuss this with anyone.

It's all politics. Estimation is a purely political game by EMs/TPMs/PMs to try and shirk responsibility for engineering outcomes.

Here's another neat tack to try in this conversation. Ask for individual examples of "good estimators," and ask for details about what makes them good estimators. You'll rarely (if ever) hear that someone was so accurate that it materially impacted a project in a positive way. What would that even look like? "I was sooo accurate that the client success people were able to say that our new product would be ready 6 months ago, and now didn't have to send a follow up email to update the timeline!!!" The only answers I've gotten are that people who are good at estimating are the ones where nobody ever really has to look at their schedule and there is no drama because they are always getting things done. This is highly contextual and rarely has to do with that person's individual estimating skill. It has to do with their project, team, EM, PM, experience relative to teammates, etc. Recently I was given an example of a client-side developer who is far more experienced than the back-end guy building his API, so he's always just sitting around waiting for the API to be finished to build his features. He almost always does 3/4 of his tasks ahead of time and then just waits for the API to be done and puts his ticket in as completed at that point. So he's not really estimating at all, and he's not accurate, it's just that the project is bottle-necked on the back-end dev speed regardless, so nobody ever cares about his estimates.

It all just seems so wishy-washy and bullshit. It's just about pushing people to work hard and get more done ("you need to hit your estimates, think of them like commitments!"). Framing this all as somehow single-handedly the engineer's problem is just a sign of someone who doesn't know how to operate a software development team effectively. Estimation is a team effort in scheduling, delegation, planning, and communication.

Why in the world (if not politics) would anyone invent a concept where it punishes someone who over-achieves and chooses to work harder for a short period of time? ("you should try to be more accurate about estimates, even if you have to slow down your work, and over working like this leads straight to burn out, be careful!) It's all just nonsense invented by MBAs who have no actual experience in anything except inane "policy" and "oversight."

Software project estimation is a real thing, but it has nothing to do with one person's ability to predict the future. It's an analytical data methodology far more than an individual, experiential skill.

[AnimalMuppet]

One place I worked said that you weren't allowed to make an estimate that was longer than three weeks. There are apparently studies showing that estimates longer than that tend to have much larger errors. So if it was going to be longer, we had to break it up into pieces until each piece was smaller than three weeks.

That could get tedious. On the other hand, we did do a lot better than normal at hitting our dates.

(I believe this shorter-than-three-weeks idea came from Extreme Programming, but I'm not quite certain of that.)

[__alias]

This is also why I like estimating tasks using the Fibonacci scale without a direct correlation to time.

In my teams, we generally set 8 points as something that would take an entire day. Every number after that jumps up in relatively large increments as they are more difficult to accurately determine

[BOOSTERHIDROGEN]

Interesting take, mind sharing how do you use fibonacci ? Thanks

[mgkimsal]

Not the OP, but one of the teams I'm on uses fib somewhat differently.

Any point with 1-2 is estimated at < 1 day. Some are literally 20 minute fixes, but ... you don't always know that up front, you just generally know it may be pretty small. 2 might sometimes go up to a day.

A 3 is assumed to be a day or two.

A 5 is assumed to be 3-4 days.

An 8 would be 1-2 weeks.

Anything higher is backlogged until it's broken down into smaller segments.

Not sure how well that compares to usages by other teams, but that's one data point for your question.

[bart3r]

https://www.youtube.com/watch?v=iZYSapFCg4A

[valenterry]

But then, you still have to connect all these 3-weeks-pieces together. And how long does that take?

No we are back to the original, overall question...

[AnimalMuppet]

The idea was that the three week pieces add up to the whole of the larger task. (Yeah, I know - only if you didn't miss anything. Take the time to think it through well enough that you don't do that. And what if you have things you don't know? Then you have to do a research project to find them out before you can give valid estimates.)

[mgkimsal]

"And what if you have things you don't know?" I usually find stuff out in the middle of work - a question comes up I don't have an answer for, and many times, no one else does either. In effect, no one can estimate it, but we didn't even know that up front. And... I've often hit things where the time to give an 'accurate' estimate takes more time than the actual work effort. Is that common in your "limit everything to 3 weeks" world?

[AnimalMuppet]

If the time to give a more accurate estimate takes more time than the actual work, you aren't dealing with an estimate longer than three weeks. If the estimate is less than a day, it's not worth getting more precise.

To your first point: Yes, that happens sometimes. When it does, your estimates can be wrong. (Hey, they're estimates - they're not prophecies.) If that happens very often, though, you might add a fudge factor for "that kind of thing". Maybe something like "unknown surprises crop up most of the time, and when they do, they take about 20% of the effort, so we'll make our best estimate, then add 20%". That won't be perfect either - sometimes it will be 40%, and sometimes 0. But, you know, estimates...

[slumdev]

> An important part of their role is estimating how long a task will take to complete

Agile exists because a very large number of people dispute this.

[groby_b]

And then jumps through large hoops to hide that it's still asking people to estimate. Sure, it's not hours, it's "velocity" and "difficulty", and you don't estimate, you play "Fibonacci Poker".

But at the end of the day, the question "can we do this in the allotted amount of time" still gets asked and answered.

What agile got right is realizing that the error bars increase superlinearly with duration, and that scope isn't fixed - so frequent estimates with frequent course correction. But you're still estimating.

[kthejoker2]

First, allotting an amount of time to delivering value is an anti-pattern in itself.

Second, Agile doesn't ask people to estimate ("respond to change over follow a plan"). Management asks people to estimate.

Jeff Patton says it best in User Story Mapping, the "client-vendor anti-pattern"

> It's the client's job to know what he wants, and explain the details to the vendor. It's the vendor's job to listen, understand, and then think through a technical approach for delivering what the client asked for. The vendor then gives her estimate - which in software lingo actually means "commitment" ..

> The real tragedy is the client understands their problem better than she's able to predict what will solve it. But in the anti-pattern, conversations about problems and solutions are replaced by discussions and agreements about requirements. No one wins.

> Try showing up at your doctor's office and giving her your "requirements". Tell her the prescriptions you'd like written and the operations you'd like scheduled. If she's nice, she'll smile and say, "That's interesting, tell me where it hurts."

> In my head, I picture a continuum where on one side is the word waiter, and on the other is the word doctor. Try to make your working relationships more like doctor-patient and less like waiter-diner.

[groby_b]

Pretty much all existing incarnations of agile have a planning meeting. And an entire edifice around managing longer-term planning. (Stories. Epics. Sagas)

Sure, we can true-scotsman, but in practice agile asks you to estimate.

In the client-vendor context, you can sidestep that with a fixed price bid. Somewhat. If you're bad at estimating your fixed price, your business will burn.

In the employee context you can sidestep that somewhat as long as you consistently deliver more value than you cost, but even then, making choices requires having an idea of opportunity cost. If you can't give that idea at all, there are usually better uses of the money.

In almost all contexts, you are compensated for your time. Almost no one likes writing blank checks.

[bottled_poe]

> Second, Agile doesn't ask people to estimate ("respond to change over follow a plan"). Management asks people to estimate.

I think it would be more accurate to say that those who pay for your time will ask you to estimate on the value which you expect to deliver in said time. That seems like a fair question to me.

[kthejoker2]

This is ipso facto, but like all creative work, they should prefer to pay you for your work product.

And I could write a book on it here (several others have!), but I think your comment points at the heart of this entire problem, namely the disconnect between the work being done and "those who pay you."

If they were truly invested in the value fulfillment cycle, they would never ask for an estimate, they would be clear about what hill they needed to be taken next in service of the product or customer (i.e. "here's where it hurts, doctor") and then help you agree on the smallest possible experiment to take the hill.

[andrekandre]

> and then help you agree on the smallest possible experiment to take the hill.

in most corporate environments, that wont cut it though... what management wants, and product owners are preassured to deliver, are large wins and overal product milestones, not incremental updates (outside or bug fixes)

> the disconnect between the work being done and "those who pay you."

yes, i think in many places, there is a fundamental tension between the "corporate thinking" and "agile thinking" and without real syncronization of methodologies and culture, any kind of "buy in from management" will usually lead to dysfunction and overall dissapointment

[icedchai]

I have fond memories of spending days in planning meetings, arguing over whether a task was 2 points or 3. Totally pointless.

[Fire-Dragon-DoL]

This is the definition of Agile, the officiak one: https://agilemanifesto.org/

[slumdev]

Story points were made up to keep project managers happy so that they leave the team alone.

This is a good indicator of how much time you should spend estimating.

[bottled_poe]

Perhaps a corollary might be that not enough great engineers exist to give good estimates, so we attempt to avoid giving estimates with much consequence attached.

[slumdev]

Great engineers are engineers who build great software.

Estimating is not and never was part of the job.

Don't be guilted into thinking you aren't great just because you don't have a crystal ball.

[eeZah7Ux]

Agile exists to micromanage people.

[slumdev]

Agile is designed not to scale.

Every framework that seeks to scale agile is ultimately a malignant attempt to retrofit it into an older method that is more conducive to reporting.