Elisabeth Pendergrass: Hello everyone and thanks for joining us today. My name is Elisabeth Pendergrass and I'd like to welcome you all to today's webinar, "Critical Cost and Schedule Target Setting with Data Driven Estimation." Before we get started, I'd like to make you all aware of a few technical items. All of you who are listening to the presentation today are currently on mute, so to ask a question at any point during the presentation, just use the q a dialog box shown on the right hand side of the webex screen. Keith will field as many of these questions as he has time for at the end of the presentation. If he doesn't have time to answer your question, don't worry. I will save all questions that we don't get to and refer them to Keith so he can get back to you. So I'll start by introducing our presenter today. Keith Ciocco has more than 30 years of experience working in sales and customer service with 25 of those years spent at QSM. As vice president, his primary responsibilities include supporting QSM clients with their estimation and measurement goals managing business development and existing client relations. He has developed and directed the implementation of the sales and customer retention process within QSM and has played a leading role in communicating the value of the QSM tools and services to professionals in the software development engineering and IT industries. So without further delay I'll let Keith get started.

 

Keith Ciocco: Great, thank you Elisabeth and excuse me thanks everyone for attending today. Really appreciate your time and hope this webinar is helpful for you. So my plan is to take you through a brief slide presentation and and to initially provide some background on QSM. And then to show some examples of how we use our tools to help with cost and schedule and quality type of types of target setting and also to show some ways that we can use these tools to help improve communication and negotiation capabilities both at the project and at the enterprise and delivery levels. Okay so let's go ahead and get started.

 

Quick background on QSM. Celebrating 43 years in business this year and we're mostly known for being the developers of the SLIM-Suite of tools, which you're going to see today or at least some of them. SLIM is an acronym for Software Life Cycle Management and we're also a big research provider and we use the research to and the tools to help our clients improve their planning and negotiating capability on their projects and their portfolio development. So that's kind of our main focus. The research actually comes from mainly from the QSM industry database. It's got over 13,000 completed projects and it basically provides the latest information on cost and schedule and effort and quality on completed projects or completed deliverables or completed project increments -- you know however you would define the work that you're doing And so studying all this data gives us a good understanding of really the fundamental relationships from project to project.

 

This is actually what we call the software production equation. It's also known as the Putnam Model. Larry Putnam Sr. actually developed these models initially and he was actually the founder of QSM. And what we're really looking at here is we're linking the amount of effort and time that it takes to deliver the value for our clients. So the value delivered is linking to the time and effort it takes to deliver it in addition to the level of productivity that it takes to deliver that work or that value or that functionality. And this is the basic model and it's really the cornerstone of the tools that you're going to see today and also really the reason that we can be so flexible in solving all these different types of estimation problems in software and technology development. And we're going to probably refer back to this equation a little bit during the webinar so I kind of wanted to show you it here at a summary level.

 

Benefits to data driven estimation #1. That's what we're going to be talking about today, data-driven estimation and how it can help us do a better job of setting our targets right and negotiating those targets. So benefit #1 is it helps us get the conversation started. #2 it helps us save a lot of money and time because it gives us the ability to improve those cost and schedule targets that we so often have to sign up for right early on. With those early decisions we want to get those targets and those goals correct, so we don't lose money in the long run and so we keep our clients happy with the deliverable for the quality and the cost and the schedule. #3 it helps to improve the communication .The data-driven estimation helps us improve communication between the management and the stakeholders. #4 it helps us improve our negotiating capability.

 

So what I'd like to do before I really get into it is I actually have a polling question and that way we can kind of get everybody involved a little bit to start. You'll approximately have one minute to answer the question and the question is: how do you currently establish your development and your delivery targets? You'll see a polling dialogue box pop up and you can choose as many answers as you would like that you think apply. And we'll go ahead and give about a minute for your answers and then we'll talk about it.

 

OK, there we go. There are your answers. So in case you can't see the dialogue, it looks like most of you chose Data-Driven Estimation as the way that you all set your targets, which is great actually. So Data-Driven Estimation was 59% of the answers. 55% of you chose Task-Level Planning. 34% chose Experience-Based. 21% chose Gut Feel. So that's really interesting actually that most of you chose Data-Driven Estimation. That's actually great, because that's we're going to talk about today, as you know. We actually find that a lot of project managers actually struggle with the early planning and so anything we can do to help improve that process is really helpful. One of the things we recommend is that, since data driven estimation allows you to have a way to negotiate your numbers early in the planning process, the data part of this helps us avoid committing to something that's unrealistic. So I'm really happy that most of you all chose data driven estimation, so most of you will be pretty familiar with some of the things I'm going to talk about today. 

 

The data part is especially helpful. We're going to show some examples of when we need to leverage that data, like if stakeholders are pushing back on the targets that we set. If we have some data-driven numbers to show them, it helps us with the negotiation and it helps keep everybody happy early in the planning process. I really appreciate those responses. Thank you again for providing those answers. We're going to continue to talk about this as we go through and show some examples of how we use the data driven estimating to help ourselves with the early planning.

 

Speaking of negotiating and using that data to help us negotiate, this is actually a scatter plot from the QSM industry database and you can see a cluster of the data here from that database. And you can see a relationship here. You can see Time on the vertical and Size on the horizontal and so this is basically saying as the projects get larger, they take longer, right? Pretty simple relationship. And then in red letters, we have what we call the "Impossible Zone." Unfortunately, the impossible zone is what a lot of software and technology development managers sign up to, because oftentimes, they don't use a data-driven estimate.They might use some some other method, like gut feel or maybe they're being forced into a schedule or a budget, something like that. So by using historical data and some modeling types of estimating tools, we can usually avoid signing up to these crazy targets early in the planning process. And that's what we're going to show some examples of today.

 

The examples are actually going to come from our SLIM Suite of Tools. We're going to be showing some examples from these tools today on how we help our clients set these targets and maybe negotiate more effectively and also maybe manage that uncertainty that often comes with the early planning. I'll give you a quick summary of these just so we understand what we're going to show today. SLIM-Estimate is our tool for running those estimates for cost and schedule and quality and being able to manage the risk on those projects and on those early decisions. It comes with SLIM-DataManager. That gives us a place to capture all that historical data. It's our repository tool. It would be if you're a client, it's your repository tool for capturing your historical data. We have SLIM-MasterPlan for estimating at the portfolio level. We have SLIM-Control for tracking and oversight. So once we either win the business, we're doing the proposal or we've decided on a on a go or no go decision, we're moving forward with the project. SLIM-Control allows us to look at plans and actuals and actually re-estimate as things change during the project or the deliverable. SLIM-Metrics is for process improvement type of analysis. It lets us see where the bottlenecks might be, if there are any, or where are the strengths, where are the weaknesses within the organization. How does one project compare to another? And how do all of our projects compare to industry data? Are we competitive? Where can we improve? That's kind of a summary of the SLIM Suite of Tools, the desktop version. Then we also have SLIM-Collaborate, which is our cloud-based software as a service tool and it basically covers the estimating capabilities that I just mentioned. So it's our cloud-based estimation tool and that's what we're going to mainly focus on today, SLIM-Collaborate.

 

So I'm going to go ahead and just pause for some questions and we're going to allow for questions at the end, as well, but I thought I might answer if there are any right now, before I jump into the details if I had any high level questions  before we get started.

 

EP: So far I just have one and that was: can these tools apply to cloud migrations?

 

KC: Yes, absolutely. These tools can apply to cloud migrations, microservices software, and application development, really any type of development or design process. So the answer is a strong yes on that. 

 

EP: What is being used to get the relative size of a project?

 

KC: That's great question. We're going to show some examples of that today. So that would be based on whatever you all determine is how you want to measure the work that you're delivering or the value that you're delivering or the functionality. So that could be features. It could be user stories. It could be high-level business requirements. So what determines it would be the user, who's managing the project. Then you can customize these tools for whatever size measure that you feel comfortable with or multiple size measures together. And we're going to show some examples of that. Actually sizing is one of the biggest challenges that people face and we have some nice ways in our tools to help model those types of challenges.

 

EP: How much data do you have in your portfolio in relation to digital transformation? And this could also be in relation this could be in regards to the QSM Database.

 

KC: I mean yeah absolutely I mean we have thousands of projects in the QSM database that relate to any type of engineering development transformation um and that can encompass all different types of technologies whether it's cloud or or or like I said microservices application development so thousands of projects that deal with transformation types of programs

 

EP: Can you use this tool with agile projects?

 

KC: Yes, absolutely. These can be configured for agile. It can be configured for more traditional types, as well, like waterfall A lot of our clients are doing agile and a lot of our clients are doing something in between agile and waterfall, kind of a hybrid. So the answer is yes. We fully support agile, as well as all the other types of software methodologies. In fact a lot of the methodologies and ideas that agile promotes, like smaller scope deliverables and smaller team sizes, are things that we see are beneficial in the data that we that we do research on. We do a lot of benchmarking and studying all the project data and a lot of the projects in our database are agile projects.

 

KC: Was that the last question for now Elisabeth?

 

EP: Yes.

 

KC: OK, sounds great and then at the end we'll leave a few minutes for more questions. 

 

So what we're looking at here is SLIM-Collaborate. And again, this is our cloud-based estimation tool or software as a service, as it might be defined. And you can see here that this could actually be your website or your web instance with your projects loaded in. You can see here that we've got the ability to set up different preferences, like email notifications and we can set up user names and passwords and things like that. You can actually set this up the way you would like it with your website name and with the people that you want to access all of your data. We have a full back office where we can set up different setup and configure different templates for all different types of life cycles. If I scroll down a little bit and look to the center bottom here, you can see a project list and these could be all of your projects either that have yet to be estimated or that have completed or that are projects that have been completed estimated and actually delivered. This is fully configurable you can see here if I go to choose Filter I can actually access some of the filters that I've set up. You can see I've got estimates that need to be approved, estimates are already approved, estimates that need changes. I've got projects that have been completed, not only estimated, but have actually been delivered and completed. I've got agile projects, waterfalls, cyber security, data warehouse, microservices, whatever you want to set this up. This would be where you could be your centralized database for your projects or your deliverables or your program increments, however you want to define the work. In this example you were looking at, my projects that have been completed and these are actually CRM types of projects and you can see we're calling them "Closeouts," because they have been completed. So really what we encourage is for our clients to capture their own historical data, but it's okay if you don't have any historical data right. I know a lot of people in the audience are actually people that do proposals, maybe for the federal government, maybe for the commercial sector, and that's totally fine. A lot of times, we're trying to win new business and we don't have any historical data and that's one of the one of the actual sweet spots of our tools - we can use the industry data and the models that are built into SLIM to help us generate reliable estimates. Also we can use these models to manage that early risk we're not really sure about: what you know, what we should, how we should price out the system. So we're going to show examples of all this. But before we start showing examples of setting our targets and using the estimates to do that, I just wanted to spend a little bit of time on the historical data part, because eventually it's a good thing to do even if you're not doing it now. I'm going to show some examples of some historical data. Then later, I'm going to show how we use that to set the targets and to also mitigate the planning risk that we go through sometimes.  

 

These again are projects that have been completed. And if I go to the portfolio dashboard here, I'm actually going to analyze that data, so I've got all these analytics that come up on my projects that have been completed. You can see here I've got all these dashboards that I can pull up. First dashboard that we're looking at is four charts. The one on the top left is a portfolio timeline, so I can see when each of these projects was started and completed. I know with the agile folks, sometimes they like to call it something different, like "deliverable" or "program increment," however you want to define it. This is when each deliverable began and ended. I've got my dollars per month that were spent. I've got my staffing full-time equivalent per month over the years. I've got my cumulative cost over the years, as well. So how much have I spent the last four or five years on my projects? It kind of gives me a kind of a bird's eye view of what's happened so far in my portfolio.

 

I can look at the data in lots of different ways. In this example here, we're looking at some pie charts. And I can say,"Ok, I had this many projects in this particular year. I can see what industry. I was mainly focused on this particular client. I was focused mainly on financial systems or financial applications. Their primary language was java and javascript." You can see they spent most of their time on major enhancements and minor enhancements. We had some new development. The quality of this data, I considered it high level of quality on most of the projects, moderate level on on the rest, and then most of these projects were actually fit into the agile or lean category. A way to look at the data at kind of a bird's eye view once everything is completed.

 

Then I can also look at some time-based performance trends. Here I'm looking at historical data. I'm looking at our Productivity Index over the years (and we're going to talk more about this productivity index measure), but in summary right now, I'll just say that it's it's a macro measure for productivity and efficiency. If you're running our QSM tools, you'll have a Productivity Index assigned to every one of your deliverables or projects. It really takes into account the overall project environment that you're working in and it's an objective quantitative measure that's calculated based on historical data. The higher this number, the better. It takes into account things like experience level, the people that were on that project, complexity of the software or the complexity of the work that you had to deliver, tools and methods you might have been using, the process that you're working and things like that. The beauty of it is it's calculated mathematically, so we're going to talk more about it in a minute. But you can see this particular client had a higher higher productivity in 2016 than all the rest of the years. And it just so happens that they actually spent the most money that year, as well. See the total effort here by year. Our staffing was also higher that year. And then we actually delivered more work in that same year. So if I'm an analyst in this company, I'm a project manager or I'm a vice president of development, I might want to go back and look at some of the details of why that happened. That way you know Y was our most productive year, also the year that we actually delivered the most work. Did we have the more experienced teams working on that work? How did the staffing profiles work? How did that save us money? Things like that.

 

Then my favorite view is looking at the historical data compared to industry trend lines and also custom trend lines. So here again we're looking at key metrics. We're looking at duration and we're looking at effort. We're looking at that Productivity Index. We're looking at staffing. And you can see here on the duration each one of these gold data points is a project that's been completed and you can see how it matches up to the trend line that we've got. This could be a custom trend line for your particular company, an internal custom trend line, or it can be an industry average. So we can compare our projects to people that are in our market space and that we're competitive with, that all built into these types of tools. You can see here we're looking at completed projects compared to either industry trend lines or our own custom trend line. Those were kind of some high-level views of historical data. We're going to show some ways that we're going to leverage that to help us set those critical targets and manage that risk early. But before I do that, I'm just going to go a little bit deeper and open up one of these projects that's been completed.

 

I had log been logged out so I'm going to go back in here.

 

I'm going to go back to my project list and we're going to open up one of these projects.

 

We're going to go into a little more detail just to show you the kind of deep history you'd like, which is which we encourage you to capture. A best practice is we want to be able to capture how long the project took. Here you can see I've done that this project started in January and ended in December, so about 12 months. We want to capture the total effort. You can see here we've done that and we've actually broken it out by phase here, as well. The total effort, total defects that were at delivery time, and the total functionality delivered or total value that we delivered. Here it was 465 story points. Now if you remember that one slide I showed that software production equation, those are those key measures. That's what we're capturing here. We're capturing how much work size, how much effort was spent, how much duration elapsed so we can calculate that Productivity Index. Here I've captured those three measures. Then when I hit "OK," it's going to bring me here and it's going to calculate that Productivity Index for that particular project. It's telling me here that we achieved an 18.8 Productivity Index, which is pretty average for a business application. 

 

If I look to my left, I can see how our effort on this particular one project compares to industry or our own custom trend line. Then we can see how it compares staffing-wise if staffed this competitively. If I scroll down a little bit, we've got another way to look at the data. We call these Five Star Reports -- Five Stars being your top 10% in the industry. This particular project performed pretty well it performed at about Three Stars, which was about top 40-50% in the industry. So overall, this project did pretty well. 

 

Now we're going to get into the core part of the webinar and show how we leverage this historical data to help us improve our estimating. And let me also just stress one more time that if you don't have any historical data, that's okay. There are no prerequisites to using these tools. We can work with what you have. A lot of the people that use these tools are doing proposals and they're trying to make very early decisions on new business. They don't have any historical data. So that's kind of a part of the value proposition of these tools is we're going to use the industry data and we're going to use the models that are built in to manage that risk and uncertainty and putting buffer into our decisions. I'm going to show you some examples now. I'm going to go to this project list again. These were projects were that were completed. Go back to the filter. Let's bring in our estimates that we've got here pending. These are our agile estimates. Someone asked about agile earlier in the webinar, so I'll bring up that set of projects here. But again, we could be supporting whatever type of methodology in software or technology development that you're working in whether, it's microservices or cloud or application development, infrastructure development, things like that.

 

Let's just go ahead and take a bird's-eye view of this data first before we get into detail. These are all estimates in my portfolio. I'm going to go to the dashboard here and I'm going to actually bring up an estimate for the whole portfolio. One way we can use that historical data that I showed a few minutes ago is to estimate and plan what we're going to spend next year. What are we going to spend in our budget? Do we have the capacity to fulfill the demand that's pending? You can see here that these are mostly estimates for my portfolio. You can see here we're breaking these out by program increment like we do in agile. We've got a Dollars per Month Estimate for the whole portfolio broken out over one year. Then I've got Staffing Full-Time Equivalent per Month and I've got my Cumulative Cost - how much am I going to need to spend next year. Now this could be for annual budgeting purposes or it could be for five or six proposals in my budget or in my portfolio. I could also be a customer evaluating vendor proposals and have vendors telling me how much this is going to cost and how long it's going to take. I can use these tools to evaluate whether those bids are realistic. We can work with both groups, but this is an example of a portfolio estimate based on historical data. I've got multiple projects, cost, schedule, effort, staffing. 

 

Now let's go a little deeper and start working on the targets that I mentioned. I can open one of these project estimates up. We want to be able to establish realistic targets early in the planning process, so we deliver something that's quality level to our client, within the right cost that we had promised and the right schedule that we had promised. So the first thing we're going to do is show you an example of actually how to set one of these estimates up. We're going to do is configure the template based on the type of project I'm about to estimate. We'll give the project a name. Here it's a program increment it's calling an Auto Program Increment. Then I'm going to set up the workflow status. For example you know where this estimate is in relation to our planning process. Is this an estimate that's ready for review? Is it a project that needs an estimate? Is it something that's already been approved? Then you remember with SLIM-Collaborate (the tool I'm showing you here), we can set up email notifications so everybody's on the same page early and so all the stakeholders are in line with what the decisions are. The stakeholders are the people that you choose to have be part of the decision making process. The next thing I'm going to do is I'm going to choose a Trend Group. These are all knowledge bases from that QSM Industry Database that I mentioned. You can see here it's broken out by all different types of application domains. When I click on one of these, what happens is it brings in the typical cost and schedule and effort and quality numbers for this type of project. If I don't have any of my own historical data, if I'm trying to make some early decisions or I'm trying to win business or I'm trying to do a proposal, I can use industry data to help me be competitive with my price quote. So I'm going to bring in the knowledge base that's relevant to the proposal that I'm working on or the early decision. If I'm not doing proposals, that's okay too. If I've just got an early decision, I'm an internal application development manager, I've got to see what should this cost, how long should it take, and bring in the knowledge bases that are relevant. I'm going to bring in the labor rates that I want to use and that can be one overall labor rate for the overall program or it can be multiple labor rates broken out by skill category or by role. 

 

I'm going to go ahead and customize the phases that are in my my project or my deliverable. You can see here I've got Concept and Planning, Requirements and Design, Development, Post Development Support. This can all be configured. I can have different names here. I'm going to decide how do I want to allocate my resources in this estimate. If I have 20 people, do they all go from start to finish or do I gradually add people on and then take people off? So I'm going to set up the model that best shows or best customizes how I allocate my resources. I'm going to pull in these configuration sets, so I'm basically going to tailor this estimate to my project environment: my phase names, milestones, reliability information. You can see here I've got a whole host of options. It can be for the DOD, package implementation, hardware design or development, agile, waterfall. Whatever type of life cycle you're working in, we can configure this, as long as it's a development type of process that you're working in. 

 

Then reliability, not only do we want to deliver on time and within budget, but we want to make sure the system is delivered at a high level of quality. So we're going to set up the parameters here that we're comfortable with, like the software needs to work 8 hours without discovering a critical defect or it's got to work a summer certain number of days a week without discovering a critical defect or a cosmetic defect or a moderate defect. However you want to set it up, the QSM tools have reliability models built in which reflects this tuning factor. This defect tuning factor that's set at 100, meaning it's set right now to the QSM reliability model. This can be tuned up or down based on your historical data. We're going to set this up based on your project environment. 

 

Then I'm going to go ahead and generate the estimates for the targets that I'm looking for. We've got multiple options to deal with this. The first option is the Feasibility option, because that's usually used by our clients that do proposals. We have a lot. In fact, I think we have some clients on today that do a lot of federal type of work, so this would be one option that you could start your estimate to do the pricing that you want to propose to your client to win the business. You can see here I've actually set up some targets. This particular client needs 25 epics delivered and they need it done in 6 months and we've got 8 people to work on it. Those are our targets that we're hoping for in this example. Of course it's flexible. I can set this up however I want. Maybe I'm not just dealing with a staffing target, I'm looking at a cost that I have to hit, maybe it's fixed price, maybe it's a certain number of person hours that I'm dealing with. So we can set our targets here to be competitive. I've got 25 epics here, but this is flexible, as well. Somebody asked about sizing earlier. I can define the functionality or value that I'm delivering however I would like, whatever fits in my particular project environment, whatever I'm comfortable with, whatever I can do consistently. 

 

I've got this other area in the tool where I can go in a lot more detail. You can see here we're actually using business use cases and interfaces and we're marrying that up to number of total epics, but I'm flexible. I can set this up for whatever type of sizing mechanism that I need. For example, maybe I'm doing a package implementation, so now I'm now I'm looking at something different. I'm counting reports and interfaces and conversions and things like that and I'm breaking them out by complexity or maybe I'm doing a data warehouse type of system that I'm proposing. So instead of features or user stories,, here I'm looking at interfaces, transformations, data structures, things like that. And QSM can help you set this up. Our support's included, so we could we could help you create a template and we can help you with your estimates if you'd like. Sizing is something that we can help you with, but a lot of times we don't have that level of detail. We're doing a proposal and and the RFP came out and said: this customer needs this at a high level, so we're going to go ahead and just put in 25 epics in this example, 6 months, 8 people. 

 

Now before I move forward with this estimate, let me just show you a couple other quick things here. I can also approach the estimate, if I don't have this level of information, we call this a "Trend-Based Method." Let's say I don't  have a specific target that I'm working to for cost and schedule, all I know is what the customer needs, functionality-wise. I'm just going to put in how many epics we need to deliver and then I'm going to let the QSM models figure out how much it's going to cost and how long it's going to take and how much effort is going to need to be spent. Another option might be our "Design to Goal" option, where we know what our Productivity Index has been in the past. We know our historical data, we know what the industry data says, we know that typically on this type of project 19 to 20 is typical on that productivity scale. We know how many people we have and we know how much functionality we need to deliver. That's just another option. Remember that Software Production Equation I showed earlier? That production equation can be adjusted based on the problem you're trying to solve. That's what we're doing here. We're looking at the estimate based on the information that you might have available. Then we have a Time Box example. This is really popular with our agile clients. A lot of times, they have a fixed price and they have a fixed schedule. So we're going to put in what their fixed schedule is and what their fixed price is or their fixed number of people can do either here and then we're going to let the QSM tools figure out how much functionality can be delivered. This Productivity Index has came from my historical data. It's based on the most recent agile projects. The SLIM-Collaborate tool here will figure out how much functionality can be delivered within those fixed parameters and it'll be able to determine whether those parameters are realistic or not.  

 

Let's go back to the initial one, the Feasibility High Level. I'm doing a proposal or making making an early decision, doesn't have to be a proposal. Time, number of people, work delivered. I hit OK and it runs the estimate for me. It's going to tell me what targets are reasonable and what targets are not. So let's look at this dashboard here for a second so you can see here that in the gold was that was my initial target. That's what I was hoping for. I was hoping to use about 10 to 12 people on this particular deliverable. QSM is saying:  you're going to need more people on this. You're going to actually need close to 20 people to deliver this on time and within budget and to be competitive. In this particular example, a lot of times it'll be the opposite. Sometimes a lot of companies will overstaff and then the QSM tools will say: you're over staffing, let's save yourself some money and let's use a smaller team. But in this example, this particular estimate is actually saying we need to budget for a few more people. You can see here on the chart, I've got my in gold were my targets that I was hoping for. I was hoping for 6 months, 8,200 person hours to deliver the 25 epics. QSM is saying, based on the models and based on the historical data, that I need to allow for a little bit more time, a lot more effort and a few more people, as well. 

 

Then if I scroll down, you can see my estimate again in gold. It's telling me that my effort estimate is moderately risky. It's within the standard deviation of the industry trend. You can see this trend line here would be the industry average. We're using this for a competitive edge. We're using industry data to say: is this proposal going to be competitive before I make a decision to go forward with it? Then you can see what QSM recommends. It's recommending in this example to allow for a little more time and effort. Then the pricing, for pricing managers that are listening, I was hoping to be competitive and maybe lowball this, but QSM is saying based on the historical data, it's going to cost me a lot more money to deliver. This is going to save me a lot of money if I know this early, because if I'm down here, it's going to be one frustrated customer later on in the process. So this is one example. These are our targets: the target I was hoping for in gold, the target QSM is saying would be more realistic.

 

Let's look at some contingency planning now. If I go to this option over here, we can actually look at some contingency plans. If I look at the Cumulative Effort Chart here, you can see in gold, which is my estimate, and this would this could be the 50 estimate right out of the box. I don't have any historical data. Then the grade region there is allowing me for some buffer. I'm saying here: give me the schedule and the budget that I've got an 80% chance of achieving or give me the schedule in the budget that I've got a 90% chance of achieving high assurance. My project is much more likely going to be in this range than in this range. So I can manage it effectively now that I know this information. I know that I'm not going to be enough resources to deliver this. I'm probably going to need to be more up in this range. Same with cost, allowing myself a higher price here. Same with staffing, a few more people. Then if I look to the top left chart, I've got this Effort Probability Chart, which gives me kind of a bigger picture saying: wow got a great chance, if we're budgeting 16,000 person hours, that's looking pretty good, about a 99% shot. If there's someone's trying to force me down here, not looking good. I can use this data now to negotiate if someone's pushing me down here. I can say, "based on the historical data, we've never achieved that before. I don't want to sign up to that, because it's going to be a deal where we lose money or we lose time or we lose quality. I don't want to sign up to this, if I only have a 10-20% chance of achieving it."

 

Now there are lots of other analytics here I can look at. If I wanted to compare my estimate with the QSM historical data or this could be based on my historical data. Maybe somebody did an estimate and they didn't use any data or maybe they did use SLIM, but is saying we need to take a second look at this based on the the most recent industry data. Here, for duration, I was hoping for 6 months. QSM is saying we probably need to allow for a couple more months. Same with the cost, like we already talked about, we're just looking at it in a different form now. We're looking at bar charts. Bar charts are great if I'm trying to pitch senior management or if I'm trying to communicate with a client. Mean Time to Defect - this is important not only do we want to finish on time, within budget, but we want the software to work well when we deliver it. Based on my targets, we were going to have a Mean Time to Defect at delivery of a little bit less than a day. QSM is saying, based on the historical data, the quality is actually going to be a little less than I was hoping for: Mean Time to Defect 0.6 of a day. Then I actually can look at total. 

 

I can look at other measures, as well. I can quickly pull up other data analytics and say: I was thinking that we'd be around 60-65 defects. QSM is saying you're going to be a lot more than that - red flag. Then our Productivity Index: I actually gave myself too much credit here. I gave myself a Productivity Index of a 21.9. The QSM historical data is telling me, based on the history, I've never achieved that high. I'm really going to be in the 19 to 20 range. I should probably be counting on more of a 19 to 20 range on productivity. Then we can go more detailed, as well, on our targets. So far we've talked a little bit about is this going to be 6 months or 12 months? Do we need 10 people or 20 now? We're going to set our targets at a more granular level. This is all built in. When I generated those initial estimates, it also broke the estimates out for me at a detail level so i can see how much money I'm going to spend by roll by month so the first chart on the left here actually breaks it out by staffing and see here the developers, the GA, the technical analysts, the scrum masters, how many people are we going to need. You can see 8 to 10 developers, how much effort are we going to spend by roll by month. You can see here the developers we're going to we need to budget, about 2100 person hours. We've got dollars per month and we've got our cumulative cost. For those pricing managers, not only do we figure out what's the overall proposal do we need to bid, but also what are we going to spend at the granular level.

 

Then we can go back to the macro, as well. Here's a quick assessment of that particular estimate. This was my target,12 people. I was hoping to finish in here, about six months, and this is the industry data trend line. It's showing me I'm pretty competitive there with my schedule estimate, probably a little quick. I probably should allow for a little more time. Here's my estimate in the chart in detail and then here's how the effort breaks out compared to the trend line, as well. I've got the duration compared to the industry average and I've got my pricing here compared to the industry average. You can see here it's a little bit lower, so if I could actually go higher on my price and still be competitive for this particular proposal. Then I can look at the contingency plans and say, give me the schedule in the budget that I've got a 95%  chance of achieving. rSo here we are again with the effort and then with the buffer built in, same with staffing, same with cost.

 

That pretty much summarizes how we would set our targets and how we would use the models to buffer that uncertainty and look at multiple scenarios using the the data-driven approach. In the end, if we can look at this information before we make those early decisions, it can really help save us big time and big money. It also can help us improve our communication and negotiation capabilities when we're working with our clients and our stakeholders. I hope today I was able to show you how we might be able to help with those early decisions. So I will go ahead and I'll hand it back to Elisabeth now if you have any questions.

 

EP: Thanks Keith. We already have a couple questions and I encourage you all to use the Q&A dialog box if you have any more. I'll start with the first one we had: I see a lot of story point projects in your example. How do you relate projects in your database with different story point size?

 

KC: Great question. So I mentioned that QSM, we do a ton of research. So we're always studying completed project data. We have a lot of sizing data and we know what the relationships are typically in the industry from one sizing metric to another. If I'm using user stories, but someone else in the company is using story points, we know what the relationship there is. We know what the typical number of story points per user story might be, same thing with other types of sizing measures. If we're using features, one group might be using features, somebody else in the organization might be using lines of code and waterfall. So we know what the typical number of lines of code per feature is in a particular application domain, so we call these gearing factors or normalization factors. We have industry data that we can use and of course we buffer in margin for error. Se use ranges, as well, that you can use when you're doing your planning. We have industry data that reflects those relationships from sizing measure to sizing measure, but also we encourage the customer to use our tools to capture their own historical data and to capture their own sizing and see what those relationships. We actually also offer a number of benchmark services where people will hire us to come out and help figure that out that kind of stuff out for them, but as part of the tools, like if you're BI. If you're using our tools, then a lot of that support is included. We would actually help you as part of the tool package to set up a sizing template. If you had an estimate for a proposal coming up, for example, we could help you with that as part of the package.

 

EP: How does the complexity of user stories affect the estimation model?

 

KC: Well it's going to affect it a lot and so we measure complexity, as well. That gets into not only the sizing area like, how complex are we defining a user story, but also that Productivity Index. What level of productivity do you typically work at for that particular type of application you're working? So that kind of question I could actually take offline. If you want to see some examples, I could actually do a demo and I could walk you through some examples. But the short answer is in the sizing template, we would we would that would reflect the complexity, as well as that Productivity Index that I've been talking about.

 

EP: Does the tool generate output that can be imported into Pro Pricer?

 

KC: Well we don't have a push button that says Pro Pricer, but we bring all our information out into Excel and I believe most of those proposal type of pricing tools work with Excel. So we would export our information to Excel and Microsoft Project. We also have a fairly robust API that we can use to integrate with most tools in the marketplace, whether they're proposal pricing types of tools or more detailed project management types of tools.

 

EP:Are there ways to correlate requirements to IU, for example SLOC, or similar?

 

KC: Yes, great question. In fact, we use IU's. We call them Implementation Units at QSM. It's one of the measures that we use a lot in our research. We know what the typical number of IU's per Business Requirement might be for a particular application domain. Our support team can actually help a lot there. We have sizing templates already built that we can you know give to our clients that have gearing factors for IU's and Business Requirements already set up. If the specific requirement isn't set up for you, we can help you set it up. We can customize the template for you, show you how to do it, that kind of thing.

 

EP: How is change request managed or captured for a future reference?

 

KC: I was actually hoping somebody would ask that question. Some of our clients deal with their change requests in the estimation tool itself and just re-estimate when things change. They update the scope and they update the Productivity Index and some of the other parameters. But we actually have a tool that automates all that called SLIM-Control, which actually provides, and I might have shown that in the early slides, that it'll actually re-estimate when things change throughout the project. So you win the business, you're tracking plans and actuals and then you're you're actually seeing how you're doing. SLIM-Control is actually running new estimates. It's actually curve fitting to what's actually happening on the project and it's re-estimating based on those changes. Actually my boss gave me this idea it's actually like kind of like a GPS for your software deliverable or your technology development. It shows you where you are and where you're where you're headed, based on the actual data. It uses these curve fitting techniques and runs projections and new estimates based on all the changes. I'm happy to show a demo of that as well if you want to connect after this.

 

EP: Can we use Cosmic Function Points with this tool?

 

KC: Absolutely yes. That's about as straightforward as I can say it. We have a full sizing capability and Cosmic Function Points is certainly one of the units that we can configure SLIM to.

 

EP: Can you limit a particular person or department's access to some of the features?

 

KC: Yes, actually in our SLIM-Collaborate tool, you can decide which people have access to which estimates or which projects or which closeout information, so you can set that up. You can set up who you want to have access to what projects and what which you know part of the tools that you want them to access.

 

EP: Do you have any insights into  story point sizing in relation to sector industry by digital transformation landscape or technology? I'm wondering if this is in relation to the QSM database. 

 

KC: The answer would be yes at a certain level. If we're working with you in support and you need to know what's the typical size measure for this type of transformation project, we can certainly go in and look at our data and try to find something similar to whatever you're working on. I would say yes at a high level. The tools themselves have trend lines and industry statistics that are built in that we keep current for you. So when you run an estimate, you'll be able to see the estimate next to an industry trend line and that's typically broken out by application domain, whether it's engineering or business appr or telecom. The support folks on our team can use their experience and actually pull out some relevant project data. Based on a particular type of digital transformation, they wouldn't be able to say this is so-and-so's data, but they could say this is the typical cost and schedule and productivity number for this type of transformation project. We signed non-disclosures and we have to keep the data at a certain level.

 

EP: Do you have data on how safety requirements impact productivity? For example DOD dash 278 for aviation?

 

KC: That particular scenario doesn't ring a bell, but we have how certain types of programs affect the quality. So we would you would say this is a certain type of DOD project, it's a weapon system or it's a command and control system and you know that project typically has this level of productivity and this uses this type of size measure and typically has this schedule on this cost type of budget. But there probably wouldn't be a button that said exactly what you just said, but we would get in the ballpark. Then the way that we manage that uncertainty would be to use the models to look at the 50% estimate compared to the 80% estimate rand then give ourselves some buffer when we're generating our estimates.

 

EP: Just to remind everyone if you came in late or if you want to share this presentation with any of your colleagues, we'll have a replay available and I will send a link to that, probably early next week. Also the slides will be available for download. If you think of any questions after the fact, please feel free to contact Keith directly or you can always contact us on our website. There's a button in top right corner of our website so you can always contact us there and we can give you more specific answers to any question that you might think of.  Thank you Keith. You did a great job answering everyone's questions. It's great to have so many questions.

 

KC: Thank you Elisabeth. I really appreciate everybody taking the time to see this and like Elisabeth said, if you have any questions feel free to reach out and also I'm happy to do some one-on-one demos or demos for your groups in particular.