How Does Agile Quality Compare?
During a recent consulting engagement, a customer asked if the QSM defect discovery model applied to Agile projects. Of course, the best (and only) way to determine this was empirically. From our database we extracted a sample of business IT projects that had completed since 2013 that recorded pre-implementation defects. 81 of these projects were Agile and 354 did not specify Agile as their development methodology. We created average trend lines for both datasets and they displayed very similar patterns that conformed to the QSM defect discovery model. This allowed us to answer our customer’s question affirmatively.
Having a large project sample at hand and being curious, we decided to compare these metrics:
- Mean time to defect (which measures the average time a system runs defect-free in the first month after implementation)
- Average development time in months
- Staffing
- Cost/effort
In a nutshell, the Agile and non-Agile projects used very similar staff sizes. The Agile projects completed sooner and expended slightly less effort. Quality was where the two project sets differed significantly. Pre-implementation, Agile projects recorded fewer defects than non-Agile ones. However, post-implementation the non-Agile projects operated longer between discovering defects in production than did Agile projects.