I am a big sports fan and since I work for a software metrics company, I started thinking about the similarities in productivity measurement in both industries. From draft picks to game planning decisions, managers in sports measure their team’s productivity to help them make better decisions in the future. Software executives and product owners do the same thing; they measure productivity in order to make better planning decisions regarding upcoming projects.

To measure productivity in baseball, we look at measures like batting average, on-base percentage, slugging percentage, and earned run average. When measuring the productivity of agile software projects, we often look at the velocity, which takes into account the number of user stories completed in each sprint. This type of historical data helps us plan effectively at the detailed level.

As part of our work at QSM, we are often asked to provide plans at the release and portfolio level. To provide these plans reliably, we use a macro level, empirically-based productivity measure that encapsulates a number of project-related factors. This measure is called the Productivity Index, an integral part of the Putnam Model.  Also known as the SLIM Model, the Putnam Model was invented by Larry Putnam Sr. almost 40 years ago and is having a big impact on software measurement more than ever today.

Once we know the total number of user stories (or any size measure), the release level effort, and the duration, we can calculate the Productivity Index of a project. The Productivity Index also takes into account the project environment including: the experience level of the team, the complexity of the software, and the quality of the tools and methods being used on the project.

The QSM team thoroughly enjoyed our time at the Gartner Symposium/ITXPO in Orlando a couple of weeks ago. We spent most of our time at the QSM exhibit, networking sessions, and at various presentation sessions.

We met many C-Level executives with process improvement on their mind. Our main focus was to learn about their specific needs in the IT budgeting and vendor management areas. We conducted question and answer sessions and provided real world examples on how to use business and software analytics to manage the complexities of IT budgeting, taking into account in-house projects as well as the project durations and costs proposed by their vendors.

Many of the senior executives we spoke with also had a need to integrate cost, duration and resource information with their project portfolio management solutions. The big focus was leveraging demand management to improve capacity planning. We provided demonstrations showing the SLIM-PPM Integration Framework which provides the release level cost, effort, resources and schedules that PPM solutions need validated to reduce risk in the enterprise portfolio.

There are so many questions around agile planning, one of the biggest being: do we need an estimate? Project managers and scrum masters will spend months developing a system either for internal use or for their clients, yet some of them say that estimates are not needed. Some recommend starting the project without an estimate. They say they will see how the first few weeks go before they generate an estimate. Others say not to worry about an estimate at all; they are a waste of time.

The problem with those recommendations is that there are business decisions that need to be made regarding whether or not to even start the project. Reliable estimates for cost and duration are needed to make these decisions. Also, for the projects that do move forward, there is usually limited information available early in the lifecycle, not enough to provide a detailed plan. Product owners need to see the big picture before a reliable detailed plan is generated.

There is also the IT manager that needs to figure out how they will allocate their resources. There is the vendor manager that needs to evaluate multiple bids for a large software system that could cost the company millions of dollars. There is the proposal manager that needs to write a proposal that must be cost competitive to win business. And, there is an annual budget at stake and the CIO needs to know how much money their development organization is going to spend over the next 12 months. You can’t support these decisions in the best way possible without reliable release level estimates.

Before an agile project starts, many product owners will run an early release estimate. Once the activities get started, managers or scrum masters begin to track the progress. When they track, they usually include the person hours of effort and the number of user stories within each sprint.  There are a number of agile tracking tools and methods in the marketplace for these tasks. 

But wouldn’t it be great if the tracking and estimation process could be combined, using the actual tracked effort and user stories to run new and improved ongoing estimates at the release level? At QSM, we have applied this process to hundreds of software projects. This type of adaptive forecasting can help save time and effort by showing when a software release is headed down the wrong path. It can also help organizations avoid signing up to inflated resource planning numbers that cause many companies to waste millions of dollars at the release and enterprise levels.

In the SLIM-Control charts below, we see the blue plans versus the red actuals and the new forecasts in white. We are capturing the total effort spent and the actual work delivered each week, then using that information to generate mathematical models that produce new empirically based forecasts at the release level.

One of the things I hear from many c-level managers is how difficult it is and how long it takes to generate reliable resource plans at the enterprise level. Many organizations take months to generate their annual budgets and often times the negotiated budgets end up being unrealistic. To fix this problem we need to combine good capacity planning with good demand management. There are a number of project portfolio management tools to help with the capacity planning. The problem is the numbers will be off if we don’t get the demand management part right.

There are world class demand management tools available that can be used by business decision makers. These tools allow us to come up with empirically based, reliable project and enterprise level resource plans. In the SLIM-Estimate view below you can see the forecasted effort by role by month.

The view below shows the plan being sanity checked with industry data so we can better negotiate our budgets with confidence.

The even bigger news is that we can automatically feed our empirically based demand numbers into our PPM tools, making the job of capacity planning much easier and more reliable. In the view below you can see two separate plans, represented in side by side columns. The original PPM plan was updated automatically from an empirically based and sanity checked plan from SLIM-Estimate.

Many of the project managers that I speak with track their software and systems projects at a very detailed level. They use detailed spreadsheets or other tools to track hours and tasks on a daily basis. This is fine, but it's important to manage the big picture so we can avoid assigning detailed tasks to duration and budget goals that are unrealistic.

By "big picture" I mean tracking at the project release level and focusing on a few key actuals: size, duration, effort, reliability, and efficiency. It's important to track these actuals to a reliable plan. These are the measures that can give us the biggest and quickest insight into a project’s potential success or failure. You can see this analysis in the SLIM-Control graphs below, showing the blue plans versus the red actuals.

Once the project is underway and we start tracking the actuals, we can generate new project forecasts based on the actual work delivered, time elapsed, and effort spent. These new forecasts are empirically-based. This will enable us to adapt to change requests, see when the project will finish and how much it will cost. The SLIM-Control graphs below show the blue plans versus the red actuals plus the new forecasts shown in white. SLIM-Control is curve-fitting to the actuals and running empirically-based mathematical models to generate the new forecasts.

We speak to a number of scrum masters, project managers, and CIOs each month. QSM does research on software development projects - all types, including agile and waterfall. We work with a huge database of completed software projects, updated with new industry data on a regular basis. We provide predictive analytics and we study cost, schedule, risk, quality, size, resource demand management, business intelligence, and vendor management.

Within the last couple of years, we have been hearing some agile managers say that there is no need to estimate agile projects. They say that they are managing in smaller increments or sprints and that they already know how much velocity can be achieved. They also say they are constantly “grooming the backlog” so there is no need for a formal estimation model to forecast the amount of work that needs to be done. They are managing at the sprint and task level so they feel like they have everything under control.

The bottom line is that agile projects still need to be estimated to see the big picture: the release estimate. It's essential to know the release cost and effort before the project plan is handed off to the scrum master and before committing to the customer. The negative feedback we hear regarding estimation is somewhat ironic because agile processes actually lend themselves very well to formal estimation since they promote the use of size measures like user stories, story points, and epics. Having good size information is a big part of estimating successfully.

Major corporations spend millions of dollars each year writing proposals to win software and systems business. They typically have a team of people that spend hours or days in strategy meetings to write what they hope will be a winning bid. Usually these companies are responding to a “Request for Proposal” which is sent out to a number of competitors. It’s almost like a sporting event. Let the games begin! Our team versus theirs. Sometimes jobs are on the line. No one wants to have to lay off people because there is not enough business coming in the door.

As part of this process, a project plan will be created. Oftentimes the team will work out a plan based on some previous experience and the opinions of a number of subject matter experts. Usually the plan will include a detailed spreadsheet with a larger number of tasks and the hours that it will take to complete each task.

The fact is most people don’t have a way to validate whether or not the plan is reliable. They also can’t see what chance they have of achieving their overall promised schedule, quality, and budget goals. They don’t have an easy way to negotiate these proposal decisions internally or with the client.

What's missing? A top-down, empirically-based estimate. Running a top-down estimate allows the proposal team to generate a “big picture” estimate for cost, duration, risk, and quality based on historical data and time tested mathematical models. With SLIM-Estimate (shown below), the team can run an overall project level estimate, and sanity-check their proposal with industry trends to make sure that their numbers are competitive and realistic.

One thing that I hear from project managers on a regular basis is that estimating a software project is a really tough thing to do. I have to tell you that it doesn’t have to be that way. Here are some reasons why project estimation can be easier than you think.

QSM is a leading demand and vendor management company. We have many years of experience working with outsource management professionals, evaluating software project vendor bids and monitoring the development progress of those bids for our clients. We are often hired to help them with their vendor management process because their past projects have failed to meet cost, effort, reliability, and duration expectations. 

It starts with the independent estimate and bid evaluation process. Our main clients are CIOs, PMO managers, purchasing managers, software project managers, and business stakeholders. Our clients will usually have a large software development or package configuration project pending. They are initially trying to figure out which vendor to hire. Vendor A will offer them a bid of 20 million dollars with a specified duration commitment and Vendor B will offer them a bid of 30 million dollars with a different duration commitment. How do we know which vendor to choose? Can Vendor A really finish with a lower cost and shorter schedule? Is the system going to work when it’s done?

The way it usually works is the client will make a decision based on their experience or gut feel. Or if they have already worked with a specific vendor in the past they will go with that vendor again based on some personal relationships that have evolved. Then the problems start. The work that was promised doesn’t get done within the promised time or the promised budget. The vendor then comes back and says they will add people to the project and everything will be ok. The client approves the revised project plan since they don’t have a way to confirm the accuracy of the revised proposal. Then even bigger problems start. More money is wasted, the schedule slips even more, and relationships sour.