Here's a risk question for you:
If today’s weather forecast predicts a 40% chance of rain and it actually rains, was that forecast “inaccurate”? If the weather channel predicts a 40% chance of rain, but the sun shines all day, was the forecast “accurate”?
Software project estimates, like weather forecasts, should always be accompanied by some explicit attempt to quantify the risk that the actual outcome will differ significantly from the estimated outcome. Estimates delivered without explicit risk assessment are more like targets: goals someone wants to achieve.
It turns out that whether it rains or not is actually a poor measure of forecasting accuracy. A 40% chance of rain forecast is accurate if, on 100 days where the forecast said 40%, it rained on 40 days and didn’t rain on 60. Likewise, the accuracy of an individual software project estimate is not determined by whether the project actually achieves its committed estimate. We can see this with a simple example: if SLIM-Estimate predicts only a 10% probability of achieving a schedule but the organization decides to commit to a plan with a 90% chance of failing, we could actually consider the estimate to be “more accurate” if the software project fails than if it is successful.
What most organizations are really looking for is not so much accurate estimates as accurate commitments, where the commitment is based on the estimate plus an appropriate level of risk resourcing. But even with best contingency planning, there is always a finite chance of “failure”—it’s just a lot less than if we don’t resource risk.
This is very similar to managing a 401(k). It is not necessary that every single investment be profitable (and one could argue that attempting to do so represents rather poor investment planning). What is necessary is that the total portfolio achieves one’s retirement goals. It is the same with projects: what is most important is that organizations manage the overall risk in their portfolio to achieve their overall business goals. And this, of course, is what SLIM-MasterPlan helps them to do.
If I leave my umbrella one day at home on a day where there is a 40% chance of rain (and the rain ruins my Armani suit) should we blame the forecast, or inadequate contingency planning? When projects commit to plans with a low probability of success, we should not be surprised if they don’t achieve their targets.
After all, the "forecast" called for pain.
Phil Armour is a QSM Senior Consultant and contributing editor on the Association for Computing Machinery’s flagship magazine “Communications of the ACM,” writing a regular column entitled “The Business of Software” where he explores issues to do with the nature and challenges of modern software with a particular emphasis on project estimation.
Read more about QSM's Project Health Check and Forecasting Services.