Early project decisions, when not much is known, are easily the hardest. They're also often the most critical. Maybe you've found yourself in a position where you need to communicate to stakeholders what your work is going to cost and how long it will take to deliver. Feeling the pressure to deliver, you might have to make decisions based on gut feel instead of past performance. This can lead to setting unrealistic targets and often results in projects going late or over budget. 

At QSM, this is when we recommend turning to historical data. Whether it's your own data or trendlines from the 13,000 validated projects in the QSM industry database, leveraging actual completed projects can make your estimates more reliable. 

Believe it or not, collecting your own project history isn't as difficult as it sounds. We recommend capturing just a few basic metrics: Functionality Delivered, Total Effort, and Total Duration. Once you have this information, you can calculate a Productivity Index, which is the measure of productivity for the overall project or release. Then all of these metrics can be leveraged by any of the other project lifecycle tools in the SLIM-Suite for estimating, tracking, and benchmarking.

In the video above, you can see how easy it is to gather your own completed projects to use early in the planning process and determine if your estimates are reasonable or not. This helps you understand the big picture before you make any important project or portfolio decisions. 

Background

No two software projects are exactly alike. So, one way to find out what a “typical” software project looks like is to take a large sample of completed projects from the QSM historical database of over 13,000 completed software projects and look at measurements of central tendency for staff, effort, size, schedule duration, and productivity.

For this study, QSM looked at validated projects that completed beginning in 2010. We eliminated 1 person projects and those that expended less than 1 person month of effort. The eliminated projects accounted for 13% of the sample. About 80% of the projects fell into the Business IT application domain, many of which were from the financial services sector. This domain includes projects that typically automate common business functions such as payroll, financial transactions, personnel, order entry, inventory management, materials handling, warranty and maintenance products. We determined both a median and an average for each metric. With the exception of schedule (project duration), these differed significantly which indicates that that the sample metric values were not normally distributed. To minimize the effect of unrepresentative projects (those that comprise a small part of the sample, but whose metric values are very large or very small), we decided to use the medians – values with 50% of the projects above and 50% below the “average” as a better measure of central tendency.

The "Typical" Project

If you were unable to attend our recent webinar, a replay is now available.

Software development managers are often in a position where they need to communicate to stakeholders what their work is going to cost and how long it will take to deliver. Unrealistic targets can be set, because decisions are made based on gut feel instead of past performance, causing projects to be late and over budget. Leveraging historical data with project planning can change all of that. In this webinar, Keith Ciocco demonstrates some of the best practices and tools that QSM uses to help clients capture and analyze historical data for better estimation, process improvement, and early decision making.

Watch the replay!

Software development managers are often in a position where they need to communicate to stakeholders what their work is going to cost and how long it will take to deliver. Unrealistic targets can be set, because decisions are made based on gut feel instead of past performance, causing projects to be late and over budget. Leveraging historical data with project planning can change all of that. In this webinar presented on Wednesday, July 14th at 1:00 PM EDT, Keith Ciocco will demonstrate some of the best practices and tools that QSM uses to help clients capture and analyze historical data for better estimation, process improvement, and early decision making.

Register now!

Although the software industry is known for growth and change, one thing has remained constant: the struggle to reduce cost, improve time to market, increase quality and maintainability, and allocate resources most efficiently. So how can we combat future challenges in a world where everything is software, from the systems in your car to the thermostat in your home to the small computer in your pocket? By using practical measurement and metrics, we can get a bird's-eye view of where we've been and where we could go, while keeping us grounded in data. Leveraging QSM's industry database of over 13,000+ completed projects, Katie Costantini takes a high-level look at changes to software schedules, effort/cost, productivity, size, and reliability metrics from 1980 to 2019. The current study compares insights to similar studies QSM has completed at regular intervals over the past four decades and answers questions like, 'what is the "typical" project over time?' and 'why are projects "shrinking?"' The results may surprise you!

Read the full white paper!

Executive teams and your end clients always want to know, “how good are our development teams?”  Agile development teams usually promise that they can deliver faster and cheaper with better quality.  But how do you truly know this is the case?  The only way to really know is to apply quantitative measurement to agile.  With the SLIM solution you can look at a completed agile project and determine the productivity that was demonstrated.  This productivity metric encompasses all environmental factors, such as how good is the skill level and experience of your development team?  How good are the tools and methodology in place?  What is the technical complexity of the software you are building? 

Developing software within the DoD presents a unique set of challenges, including but not limited to budget cuts, Congressionally mandated changes, changing software requirements, and so on. It should come as no surprise, therefore, that cost estimators have faced significant challenges when estimating systems in the Defense arena. A recent initiative put forth by the DoD was to improve its estimation process by leveraging historical data collected from forensic analyses of recently completed software development efforts. This article by Taylor Putnam-Majarian and John Staiger, discusses (1) some of the challenges faced throughout this initiative, (2) the data collection process, and (3) how one can leverage data to improve cost estimates. This article was originally published in Crosstalk Magazine.

Read the article!

Managing vendors has become increasingly important in recent years.  In my account management role at QSM, I see both sides of the vendor management relationship.  The client wants a proven vendor that will partner with them in achieving their IT goals; and the vendor wants to win that business, employ their workers, and hopefully earn more work.  Unfortunately, that state of client/vendor Zen is not often achieved, usually due to legitimate (and sometimes not) misunderstandings on both sides.

On the client side, they are concerned with selecting a vendor with whom they are confident their tasks and deliverables will be achieved on time, within budget, and of the best possible quality.  After a round of RFI’s, then RFQ’s, then a final down select process, the vendor is chosen and work begins.  Often, at least in my experience, the overriding decision criteria comes down to cost, which makes sense, to a degree.  But in many cases, cheapest, I mean, least expensive bids often rule the day.  This kind of decision-making comes with its own set of risks; the most obvious is you get what you pay for and it’s often an ill-prepared vendor.

“No, we don’t have any historical project data collected” is the statement I hear with some frequency when speaking to organizations about their IT project estimating processes.  Ideally we use client history to calibrate and tune the project estimates we provide.  In my quest to spread the word about parametric estimating I often encounter this notion that organizations don’t believe they have historical data in a retrievable form.  In almost every case that I have been involved, it turned out that the historical data was present, just not in the form of a 1,000 rowed spreadsheet.  Often times the data is more available than the client is aware.

Our approach works at a macro level so we are seeking overall project metrics of cost, schedule, size, staffing and defects.  If the actual formal documentation of history is not available for these five core metrics, then it usually is available by leveraging various sources within the organization.  We have found it’s common to resurrect a project’s outcome by seeking feedback from the team that worked the project, however if that’s not possible due to attrition, re-org or other disrupting factors, we can usually find the project metrics through other means.  Those other means may be time and defect tracking tools, requirements analysis tools and accounting systems.  The data is almost always documented somewhere.   

Dear Carol: 

Given all your international experience, I’m hoping you can tell me where I can find a large, freely available industry database that project managers could use for software estimation and/or benchmarking.  After 5 decades of software development wouldn’t you think that we could put together a software estimation or benchmarking database that the world could use for free? 

- Hopeful in Hartford

Dear Hopeful:  

Great question – and the dream of many IT project managers.  It might seem like an easy concept (just collect actual effort and project size and use it for future estimates); in practice it’s not that simple.

What I know is that in software estimation and benchmarking, there is no free lunch -- you get what you pay for.  And I’ll explain why…