For more in-depth information and research from this database, check out the updated 2023 QSM Database in Depth whitepaper and the QSM Software Almanac: 2019 Edition.
The QSM database is the cornerstone of our business. We use validated quality and performance data (size, time, effort, defects, and productivity) collected from over 14,400 completed software projects to keep our products current with the latest tools and methods, to support our benchmarking business, to inform our customers as they move into new areas, and to develop better predictive algorithms.
Data Sources
Since 1978, QSM has collected completed project data from licensed SLIM Suite users and trained QSM consulting staff. Consulting data is also collected by permission during productivity assessment, benchmark, software estimation, project audit, and cost-to-complete engagements. Many projects in our database are subject to non-disclosure agreements; but regardless of whether formal agreements are in place, it is our policy to guard the confidentiality and identity of all data contributors. To preclude identification of individual projects/companies or disclosure of sensitive business information, we release industry data in summary form only.
In 1994, QSM began collecting project data continuously, updating the database every 2-3 years. Over the last 5 years, we have added an average of 300 validated projects each year.
Data Quality
Only projects rated Medium or High confidence are used in QSM’s industry trend lines and research. Before being added to the database, incoming projects are carefully screened. On average, we reject about one third of the projects screened per update.
Data Metrics
Our core metric set focuses on size, time, effort, and defects for the Concept Definition, Requirements & Design, Code & Test, and Maintenance phases. These core measurements are supplemented by nearly 300 other quantitative and qualitative metrics. Nearly all (99.9%) of our projects have time and effort data for the Code & Test phase. while 72% have time/effort data for both the R&D and C&T phases. By contrast, only 37% of projects report time and effort data for the Concept Definition and Maintenance phases.
Delivered size data is recorded for 97% of projects in our database, with defect data reported for 44% of projects.
Industry Data
QSM data is stratified into 9 major application types (Avionics, Business/IT, Command & Control, Microcode, Process Control, Real Time, Scientific, System Software, and Telecom) and 45 sub-types. Real Time, Avionics, and Microcode projects typically exhibit the lowest productivity, taking longer to develop than Business/IT projects that automate well established and/or less algorithmically complex processes, though this is changing somewhat due to the increasing complexity of the target platforms used by modern IT programs. Software projects predominate, but we have a growing number of hardware and infrastructure (non-software call center) projects as well.
Data contributors include DoD, civilian commercial firms, and national, state, and local government entities. In addition to application complexity bins, our data is also broken out by major industry and industry sector. Major industries include the financial sector, banking, insurance, manufacturing, telecommunications, systems integration, medical, aerospace, utilities, defense, and government.
Methodology Data
The QSM database includes a variety of lifecycle and development methodologies (Incremental, Agile, RUP, Spiral, Waterfall, Object Oriented) and standards (CMM/CMMI, DoD, ISO).
Language Data
Over 700 languages are represented with most projects recording multiple languages. Common primary languages are JAVA, COBOL, C, C++, C#, VISUAL BASIC, .NET, IEF / COOLGEN, PL/1, ABAP, SQL, ORACLE, POWERBUILDER, SABRETALK, JAVA SCRIPT, DATASTAGE, HTML. Frequently used secondary languages include JCL, SQL, JAVA, COBOL, ASSEMBLER, C++, HTML, VISUAL BASIC, XML, ASP.NET, and JSP.
Global Data
QSM has collected and analyzed software projects from North America, South/Central America, Europe, Asia, Australia, and Africa. About 54% of our data is from North America. Another 34% is from Europe, with the remainder developed in South America, Asia, Africa, and Australia/New Zealand.
This SLIM-Metrics chart shows Productivity vs. delivered System Size for completed IT, Engineering, and Real Time Group systems. Because they are the most complex, Real Time group projects (black dots) have the lowest productivity, with Engineering group projects (purple) in the middle and IT systems (green) exhibiting the highest productivity. QSM stratifies project data into homogenous subsets to reduce variation and capture the innate behavioral characteristics of different software application domains.
QSM Industry Trend Lines
QSM industry trend lines are available for nine, high-level application types, six application subgroups, and three application super groups. The nine application domains are:
| Business | Command & Control | Scientific |
| System Software | Telecommunications | Process Control |
| Avionics | Microcode/Firmware | Real-time Embedded |
Stratifying the data by application type reduces the variability at each size range and allows for more accurate curve fitting. One application domain, Business/IT projects, has been further stratified into several sub-groupings:
| Business Agile | Package Implementation |
| Business Government | Web Systems |
| Business Financial | Business Cloud Migration |
Three application super groups are also available to benchmark projects of mixed or unknown application domains:
| Real Time Group | Engineering Group |
| All Systems |
There are several ways QSM clients can access the QSM database:
- Our SLIM Tools Contain Current Industry Performance Trends: QSM provides up to date trends for 17 application types with QSM software project estimation and benchmarking tools.
- We Answer Your Basic Questions: Our support team is happy to answer basic questions about the QSM database. Let us do the research to answer your estimation and benchmarking questions! We can provide graphs and summaries that allow you to compare your projects against both industry trend lines and actual projects that are similar in size, application type, and complexity. Basic questions usually involve research that we can accomplish in less than 4 hours. Note: more extensive research is available through our consulting group.
- We Provide Benchmarking Services: QSM’s benchmarking service can help you assess your current levels of productivity and quality, identify exceptional or underperforming projects, analyze the root causes of poor performance and provide a roadmap for improvement, and build a business case for process improvement initiatives. Our benchmark service will position your projects against relevant industry data and measure your variation from the norm.
- We Provide Client-Directed Research: QSM can be engaged to perform research and answer specific questions about performance, cause and effect relationships, or the impact of various tools or practices on software development projects. Current research from our database analysis is also available for download from our website and blog.