Software project benchmarking and estimating leverages the power of historical project data to do solid project estimates, yet the concepts behind such processes are often not well understood. Benchmarking and estimating rely on productivity comparisons with completed (actual) projects in a historical database and on parametric equations that mimic real life. I find that technical concepts such as software estimation or benchmarking often can be explained by using analogies that work in other industries. As I was thinking about benchmarking and estimating this week, the popular children’s book, Dr. Seuss's Green Eggs and Ham, came to mind.
I was talking about data mining, benchmarking, and the SLIM Suite of software estimating tools with QSM’s research director, Kate Armel. It seems that many project estimators believe that creating microscopic slices of project data is the key to precision in estimating and benchmarking, when, in reality, bigger chunks of data take less time to assemble and provide greater value. Projects are never exact duplicates of each other, however, there are valuable trends and patterns that come out of a few common characteristics.
Consider Green Eggs and Ham where the main character, Sam I Am, tries to convince a friend to try something novel: green eggs and ham. After a lengthy back and forth discussion, sort of a slice and dice conversation, the lead character finally prevails.
The friend finally gets tired of the cajoling, culminating in:
I do not like them here or there.
I do not like them ANYWHERE!I do not like
green eggs
and ham!
Of course, for Seuss, the exchange IS the story, and it depicts just how difficult we sometimes make things for ourselves.
I believe that we, as project managers and customers, often do the same thing. When we’re looking to do software estimating based on “similar” completed projects, we go overboard trying to line up every characteristic we think needs to be taken into account. Like the friend in Green Eggs and Ham, we can save ourselves time and energy (and get better results) if we identify and focus on the major productivity factors. There are a few major project factors that affect project delivery and team productivity, and a great many that do not.
As quantitative professionals, we often believe that software estimating and benchmarking accuracy increases when we match up every variable, instead of concentrating on the primary few. As Kate Armel explains: “the more finely you slice and dice the data, the smaller the resulting “pieces” (sample sizes) become. The problem with small sample sizes is that they are inherently more variable in their behavior over time, so it becomes harder to tell whether the changes you’re seeing are real, or merely reflect the outsized influence outliers have on very small samples. Most companies use averages rather than medians to measure central tendency. Even the median will fluctuate more with small samples, but the average is even more variable!”
Research shows that there are really only a handful of primary productivity drivers that markedly influence the estimating outcome (cost, effort and schedule) of a project. Unlike a Veg-O-Matic (a popular infomercial product that would finely dice vegetables circa the 1960’s), using trend lines and following what the data tells us yields superior results to guessing at what we, as project managers, think would make a difference.
Primary Productivity Drivers
At the same time that we know that software projects involve creative design and innovation, history proves that much of software development is predictable and patternistic. Project estimating based on statistical analysis (such as the Monte Carlo simulation used in SLIM-Estimate) that considers the major productivity drivers deliver the most accurate and realistic estimates.
Certainly there are anomalies that arise to challenge even the best estimating models, however, factors like these don’t affect the majority of software projects. Historical databases prove that projects follow trend lines based on productivity factors. Even when a software product is so unique and different that customers and programmers feel challenged, the software development usually follows repeatable life cycle tasks that can be scaled and modeled given a team’s past performance.
QSM researchers consider the following list of factors as the most important productivity drivers (factors that have an influence on project duration, cost, and/or effort).
- Application type and domain (“what it does” – a proxy for complexity. Think IT vs real time software);
- Average team size (ranges of team size);
- Programming language level (2GL vs 3GL vs 4GL Think power tools versus hand tools);
- Hardware platform (and how many different platforms);
- Project type (new development or enhancement based on amount of reuse and difficulty of modifying poorly written or documented legacy systems).
This list is similar to the one developed during the 2008 PPE project at the Software Engineering Institute in conjunction with QSM and other leading software estimating experts.
Slicing and dicing data beyond these major factors leads to unnecessarily small sample sets (smaller number of “comparable” historical projects) and reduces the reliability of the resultant estimate. The more similar projects an estimate is based on, the less variability comes into play. For example, if I try to line up every possible project factor between a database and my project, I might only have one or two projects that are comparable. I know I’m not going to base my project estimate on only two comparable projects – I’d rather rely on ten or fifteen completed projects that follow a similar pattern. Two data are unlikely to identify the correct “trend line.” Not only will my estimate be more realistic, but I’ve saved myself time in the process without reducing accuracy.
What does this all mean to you?
As a project manager, software estimating and benchmarking just got a whole lot easier AND more reliable at the same time. I can use SLIM to develop a reliable software project estimate quickly and easily simply by inputting the key productivity factors above, and I don’t have to worry needlessly about the myriad of potential other (less significant) factors. I can also quickly test out alternative project approaches (like varying team size or stretching out the schedule) and see how the estimates change. And, at a glance, SLIM will tell me if my project and the constraints I set (like schedule compression, functional size, maximum team size, etc.) creates an impossible project. Knowing ahead of time that my (or my management’s) optimism is overzealous is a beautiful thing.
While some customers take solace in thinking that project estimating gets better with a data Veg-O-Matic or after a Dr. Seussian discussion, I am relieved to know that my estimates are even better when I approach estimating armed with robust sample sizes and proven productivity drivers. It’s refreshing (and reassuring) to note that sometimes the straightforward approach leads to better results.
And, in the words of Seuss, “I like it, I like it, Sam I Am.”