Part III: The Caveats

In Part 1 of How Much Estimation? we noted that there is an optimal amount of time and effort that should be spent in producing an estimate based on the target cost of a project and business practice being supported.

In Part 2: Estimate the Estimate, we saw that the formula to calculate this optimal time (as measured at NASA)  calculates the Cost of Estimate as the Target_Cost raised to the power 0.35 (approximately the cube root of the Target Cost).  The factor that defines the business practice (either by early lifecycle phase or perhaps by the “expected precision” of the estimate) is a linear factor ranging from a value of 24 to a value of 115.

Those Caveats!

I mentioned that there were caveats with the calculation.  Here they are:

• Assuming difficulty is a function of size–we make this assumption all the time in estimation.  Larger projects (in both size and cost) are more complex and therefore more difficult.  In estimating the estimate, we are assuming that two projects of the same target cost have the same difficulty.  While this is a reasonable assumption, it might be that one project has much better early data than another (say for a project replacing an old system versus one developing something quite new).  The quality of the data might require more (or less) work to produce a defensible estimate.
• Discrete stages of estimation–the model defines only three types of estimate: concept, budgeting and planning.  We might want to invest in a more detailed budgetary estimate, or a less detailed concept estimate (say for a “blue sky” project).
• Level loading the estimate work–this is something we rarely do in projects.  If I have a 169 SHrs effort to produce an estimate, can I get it done in one hour with 169 people?  How about half an hour with 338 people?  Clearly not.  
• Work time versus elapsed time–just because we have three people available to work half of each 8 hour day does not necessarily mean that they can get through the work in the total effort divided by the number 12.  If data is not available, or if there are weekends or holidays, if essential SMEs cannot take a meeting until next week, then the calendar time will be different.  Just like a real project.
• More important–some projects are just more important irrespective of target cost.  If the project is strategic, for a very important customer, or the apple of the CEO’s eye, perhaps we should spend more time on it.  If the project is a likely non-starter, irrespective of target cost, perhaps we should spend more of our effort elsewhere.
• What if the Target Cost is wrong?–what if the initial projected cost (which was only a goal) turns out to be quite wrong and not achievable?

Adjustments

I have found the following practices helpful in both adjusting for the caveats and helping to defend the model and make it more acceptable in operation:

Slices

At two clients that applied this process we introduced the concept of “slices” (though I don’t remember who came up with the name).  Each calculation represented one slice.  So for the $10m example I used, a “one slice” Planning estimate would take three people about 27 days and cost around $32k.  While most estimates would conform to this model on occasions they would deliberately choose to fund a “two slice” estimate (for those strategic or CEO projects) or a “half slice” estimate (for the blue sky, unlikely projects).  While this was mostly done for strategic projects, we also did it occasionally for projects where we though the estimate would be “more difficult” and the value warranted the extra cost.

Sliding Estimates

Instead of having three values, I created a tool that used a slidebar to locate an estimate “between” (say) Budgeting and Planning.  This allocated more resources than a straight Budgeting estimate but not quite as much as a Planning estimate.  Along with the slices, this gave plenty of flexibility while still keeping a consistent process.

Level Loading

We don’t normally do this on projects.  However, SLIM-Estimate’s Level Load model actually works well for (a) small projects with (b) few people over (c) a short timeframe.  And that is what an estimate is.  As long as we don’t get silly with adding people, it works well.

Target Cost of Project

So what if the initial Target Cost turns out to be way wrong?  This is quite normal.  I might start with a Target Cost of $10m.  Allocate my resources and produce a conceptual estimate (in 6 days at a cost of $6,764) only to produce an estimate that says this project is more likely a $15m project.   This is where the business practice comes in.  The $10m was a goal not an estimate.  The $15m estimate still carries significant uncertainty (and SLIM-Estimate will tell us what that is) and the business decision will be: given the likely cost is $15m and the uncertainty on that $15m also has a cost–should we continue funding this project and at what level?  $10m with a very good chance on not achieving it?  $15m with a 50:50 chance?  More cost with lower risk?

This clearly shows the difference between three values that organizations often conflate:

• The Target–what we would like to have happen
• The Estimate–what is most likely to happen given what we know now
• The Commitment–what we should allocate to deal with the most likely situation plus the risk that the most likely situation is incorrect.

The advantage of using the Cost of Estimation approach is that the estimation activity itself is intentionally bounded stopping both the “drive-by” and the “never-ending” estimate.  It also has the additional plus of getting people to think more in terms of the cost versus the value delivered.