During a recent consulting engagement, a customer asked if the QSM defect discovery model applied to Agile projects. Of course, the best (and only) way to determine this was empirically. From our database we extracted a sample of business IT projects that had completed since 2013 that recorded pre-implementation defects. 81 of these projects were Agile and 354 did not specify Agile as their development methodology. We created average trend lines for both datasets and they displayed very similar patterns that conformed to the QSM defect discovery model. This allowed us to answer our customer’s question affirmatively.
Having a large project sample at hand and being curious, we decided to compare these metrics:
- Mean time to defect (which measures the average time a system runs defect-free in the first month after implementation)
- Average development time in months
- Staffing
- Cost/effort
In a nutshell, the Agile and non-Agile projects used very similar staff sizes. The Agile projects completed sooner and expended slightly less effort. Quality was where the two project sets differed significantly. Pre-implementation, Agile projects recorded fewer defects than non-Agile ones. However, post-implementation the non-Agile projects operated longer between discovering defects in production than did Agile projects.
Why Agile projects discover fewer defects than non-Agile ones prior to going live and deliver with lower initial quality in production requires more research and analysis than has been done here. Here are a couple of theories that should be analyzed empirically:
- Agile projects tend to complete more quickly. When project schedule is compressed, testing is often shortchanged. Is this occurring?
- Agile is more output/delivery oriented while quality is a process focused endeavor (a generalization). Does this difference in focus contribute to the differences in pre- and post- implementation defects between Agile and non-Agile projects?
All we can definitively say now is that based on the project data we have examined, more defects appear to slip through into UAT and production in Agile projects. For organizations using or contemplating using the Agile development methodology, this suggests that testing and software quality require more focus than they generally receive with Agile methods. Problems found in production are more expensive to correct, and more embarrassing.