[knucklesandwich]

Here's the problem I keep coming back to with task estimation for implementing software. This is the absolute minimum description of what a software engineer's responsibilities are: Giving the correct set of instructions to a computer about how to accomplish a task.

Given that, a precise understanding of the time involved to accomplish a task requires knowledge of the time it takes to come up with the correct instructions + the time it takes to give the correct instructions to the computer. The time cost to give instructions to a computer is effectively linear in the size of those instructions (its whatever my typing speed is times that size), but the time cost to understand what those instructions need to be is effectively unknowable unless I've already got those instructions. It's sort of like the halting problem for people... if I knew how long it would take to come up with a reasonably precise version of them, you'd have a version of those instructions that could execute in some language (say some imaginary pseudocode language), or I'd have determined that the task is impossible.

I can keep a heuristic understanding of how long a task will take based on previous tasks I've completed (i.e. this task sounds similar to this previous one, which took me this long. I'll use that for my estimate), but there will always be some set of tasks for which I'm wildly wrong unless I precisely examine what's involved.

If it's better to estimate in a fashion such that I deliver everything I promise (and that we can have a planning session that doesn't last the entire sprint), I can pad my estimate by a large factor. In general, its easier to meet the criteria for this situation, but there will still be tasks for which my estimate can be wildly off. This can also lead to unhappy product owners who assume I'm trying to rig planning to excuse less work.

How do you come out of this exercise unscathed and with valuable information?