As the Apps Global CTO for Microsoft Services, Lenny Fenster sees the need for estimation in many shapes and sizes throughout the world. In his twenty years at Microsoft, Lenny has also seen many different attempts to improve how Microsoft Services estimates time and effort for software development projects. Not all of them have hit the mark. In this presentation for the QSM 2018 Virtual Conference, Lenny talks about the evolution his team is driving in Microsoft Services to improve the maturity, consistency, and defensibility of software estimation for some of the largest and most complex software projects in the world. He talks specifically about the intentional separation of scope and estimation and the use of SLIM as a key ingredient in the success they are now having. Estimates are now done much quicker, reducing the time to run an estimate from days to just 4.5 hours.
Lenny was gracious enough to answer questions throughout his presentation about the estimation process at Microsoft Services. This sparked great participation from our audience, who asked a number of questions worth resharing. Here are the highlights:
Q: Did you experience any resistance among the architects in changing the way they did estimation to a new approach?
A: We did. It actually was the biggest challenge. What we did here was we took control away from the architects, a little bit of the control. And the biggest challenge I think we faced was, and I see this with our customers because we go out to customers and we talk about this and promote to them this way that they can do this as well is an adoption in change management. Initially, although the architects didn't want to do the estimate, they also had control over what they said the time and effort was. And what we said is, we want you to focus and so no longer will you need to think about what the time and effort are. That's the math. So we read Larry Sr.'s book and really digested the algorithm for giving scope or size, productivity and you get your time and effort on the other end. And we used that to intentionally separate the roles and responsibilities. So quality and productivity we want to get from our actuals and we're going through the same process that Angelo [Moore] talked about with mapping them back, but the idea to get our productivity and our quality from the actuals given different kinds of pivot points. Scope is where we want our architects to focus. We want them to be architects. We want them to focus on the architecture. It is important for them to do the requirement analysis. It is important for them to decompose those down. It is important for them to size them from a complexity perspective. But what we don't want them to do anymore is tell us what you think the time and effort should be. We're going to have other people that do that, because that's the math and that's what SLIM helps us do at the end of the day is to tell us what the time and effort is, based upon all that. So that's the biggest challege for some architects, although they love doing the estimation, they also had control over it. And starting to see that this was something else doing that for them and adopting that was probably the biggest challenge we had. It didn't last long. It would usually only take one or two instances for them to see the benefits and how they could actually focus on why they actually got into the job in the first place. They didn't become architects so they could do estimates. They became architects to be able to work with customers, to be able to focus on technology, to do the solutioning. The requirement is be able to decompose the solution down, but they didn't become architects to do the estimation part of it. That was thrust upon them, because they were the closest to it.
Q: Are you no longer using a bottom-up estimating approach, in combination with the top-down/parametric SLIM-Estimate?
A: We have standardized on a top-down approach. We used to do a bottoms-up approach. It used to take a long time. We actually, in our conversations with QSM, we tested out our theory of where we would spend a lot of time doing a bottoms-up approach. It was much more costly to do a bottoms-up approach and we tested that against someone coming in sight unseen and doing the same thing from a top-down approach and got within 2% I think of what the result was. So after doing that a couple times, we were pretty convinced that the top-down approach came close enough for us to be able to take that. And I again, it's a lot quicker and less costly to do it that way.
Q: Did or do you get pushback from the #NoEstimates agile developer mindset? Any hints or statements to overcome such resistence?
A: I would say that there's a lot of differing points of view on agile and there are many folks out there that feel agile means that one can have less discipline. We are very much the opposite of that. In fact, with an agile-driven approach we believe it really requires much more discipline for one to do that. There's a lot with agile and the "no need to estimate" that's just one of the things we hear with some of the differing opinions from an agile perspective. We respectfully disagree. It's not an agile thing. I think at the end of the day what we're trying to do is get further down the cone of uncertainty and that's methodology agnostic. We're not all the way down the cone of uncertainty, so we actually have other things that complement this like Sprint Zeros and stuff like that where we still use the EFU methodology and will estimate again at the end of the Sprint Zero after we have a better idea of the product backlog. But it is very rare where our customer just wants us to come in and not tell them how much time we think it will take or how much we think it will cost. We probably get more pushback on the other side where they want to know the things before we even have a better idea of the scope.
Q: I'm an Architect (an EA, and a Solution Architect too), and an Estimator too. As was said earlier, the challenge is sizing. By this approach, the Estimator role as a SME is becoming a commodity. They should rely on the features of the good tools provided by QSM, for example. What are your thoughts?
A: The architects don't use QSM. The estimators use QSM and what we are guiding strongly is that the architects aren't doing that. What we want the architects to do is they're working with the customer, understanding the problem, creating the solution, both in terms of technical, as well as decomposing the requirements down to the epics, features, user stories, and understanding the size and the complexity of those user stories. Their main role is to understand those aspects from it and that gets moved over to the estimators that then take that scope and leverage SLIM to tell them what the time and effort are. There are other tweaks that are involved as well. We'll look at things like, if a team's been involved there before, the architect will capture those things in working with the customer, if it's a newer technology - other things that we'll include on top of EFUs to help when working with the estimator to tweak the PI, given the bells and levers and all that stuff the estimator knows but the architect doesn't need to. In fact, I was giving a similar presentation internally the other day and someone asked me, "well are we going to just trust what comes of SLIM?" and I said, "let me rephrase that: we do trust what comes out of SLIM. It's not going to. We do trust it." Now it's a conversation. They are iteratively working on it, but that is absolutely core to this.
Q: Do the engagement partners buy into the estimates?
A: Not everybody's read the book, The Five Core Metrics. That same adoption and change management that we see internally, I would say we see with partners and customers. It takes a little bit of evidence to show that. I'll give you an example. Microsoft's a huge organization and we have different teams that work on different things and so we're working with an engagement partner that's focused on a specific industry and we were working with them to create an EFU template, because a lot of it is dev so we were the primary folks that got engaged with this specific industry. So we made a commitment to them. We said, let's get in actuals and we did that. We got the actuals and we said, let's run it through our tools. Let's run both of them through and my commitment to you is that if your method comes out closer to the actuals, not the estimate, but the actuals we saw, then we'll keep working on ours, but if ours is closer, then we should use the new approach, and ours ended up being closer to the actuals. I haven't seen engagement partners just willingly say, "yep, ok, that's great," unless they know SLIM and we have seen that. They say, "you're using SLIM. That's great. We get it." But if they haven't and it's new to them, it requires a little bit of evidence for them to get over that hump.
Hear more of the Q&A in the full webinar!