In Part 1 of How Much Estimation? we noted that there is an optimal amount of time and effort that should be spent in producing an estimate based on the target cost of a project and business practice being supported.

In Part 2: Estimate the Estimate, we saw that the formula to calculate this optimal time (as measured at NASA)  calculates the Cost of Estimate as the Target_Cost raised to the power 0.35 (approximately the cube root of the Target Cost).  The factor that defines the business practice (either by early lifecycle phase or perhaps by the “expected precision” of the estimate) is a linear factor ranging from a value of 24 to a value of 115.

Those Caveats!

I mentioned that there were caveats with the calculation.  Here they are:

In Part 1 of How Much Estimation, we observed that both too much time and effort and too little time and effort spent on estimating are less than optimal.  Combining:

• The cost of producing an estimate—which is a function of the number of people working on the estimate and how long they work
• The cost of variance in the results of the estimate—that is, how much the estimate varies from experienced actuals and what that variance will likely cost the project.  This is typically a function of the number of unknowns at the time of estimating for which the project cannot easily adjust and which will require additional unplanned resources of time, effort, and staff.

We get a U-shaped curve, at the bottom of which is the optimal time: we’ve spent enough time and effort to minimize the sum of the cost of estimate and the cost of variance.

The question is: how to calculate this point?  It will not be the same for a very large complex project and a very small simple project.  Also we don’t want a complicated and time-consuming approach to calculate the cost of estimate—it should be quick and simple.

NASA’s Deep Space Network (DSN) projectⁱ developed a mechanism for this calculation based on two simple parameters:

Target Cost of Project

This is goal cost of the project as first envisaged in the project concept.  It is NOT the estimated cost of the project (which hasn’t been calculated yet).  Projects for which we expect and plan to spend a lot of money should clearly have more time and effort spent in estimating simply because more is at risk.

How much time and effort should we spend to produce an estimate?  Project estimation, like any other activity, must balance the cost of the activity with the value produced.

There are two extreme situations that organizations must avoid:

The Drive-By Estimate

The drive-by estimate occurs when a senior executive corners a project manager or developer and requires an immediate answer to the estimation questions: “When can we get this project done?”  “How much will it cost?” and “How many people do we need?" (the equally pertinent questions: “How much functionality will we deliver?” and “What will the quality be?” seem to get much less attention).

Depending on the pressure applied, the estimator must cough up some numbers rather quickly. Since the estimate has not been given much time and attention, it is usually of low quality. Making a critical business decision based on such a perfunctory estimate is dangerous and often costly.

The Never-Ending Estimate

Less common is the estimation process that goes on and on.  In order to make an estimate “safer” an organization may seek to remove uncertainty in the project and the data used to create the estimate. One way to do that is to analyze the situation more and more. Any time we spend more time and more effort in producing an estimate we will generally produce a more precise and defensible result. The trouble is the work we have to do to remove all the uncertainty is pretty much the same work we have to do to run the project. So companies can end up in the odd situation where, in order to decide if they should do the work what resources they should allocate to the project, they actually do the work and use up the resources.

Here's a risk question for you:

If today’s weather forecast predicts a 40% chance of rain and it actually rains, was that forecast “inaccurate”?  If the weather channel predicts a 40% chance of rain, but the sun shines all day, was the forecast “accurate”?

Software project estimates, like weather forecasts, should always be accompanied by some explicit attempt to quantify the risk that the actual outcome will differ significantly from the estimated outcome.  Estimates delivered without explicit risk assessment are more like targets: goals someone wants to achieve.

It turns out that whether it rains or not is actually a poor measure of forecasting accuracy.  A 40% chance of rain forecast is accurate if, on 100 days where the forecast said 40%, it rained on 40 days and didn’t rain on 60. Likewise, the accuracy of an individual software project estimate is not determined by whether the project actually achieves its committed estimate.  We can see this with a simple example: if SLIM-Estimate predicts only a 10% probability of achieving a schedule but the organization decides to commit to a plan with a 90% chance of failing, we could actually consider the estimate to be “more accurate” if the software project fails than if it is successful.

What most organizations are really looking for is not so much accurate estimates as accurate commitments, where the commitment is based on the estimate plus an appropriate level of risk resourcing.  But even with best contingency planning, there is always a finite chance of “failure”—it’s just a lot less than if we don’t resource risk.