How can I quantitatively evaluate potential suppliers and determine whether or not a potential supplier is likely to perform as promised?
Railtrack is the company responsible for the railway infrastructure in the UK. It is implementing a very large program of structural and system improvements, one of which is the provision of a new communication system, “DART”, for train drivers. The system will use a public digital mobile phone network and will maintain a real-time train location database updated both by GPS and by existing railway systems. The contract to build this system has just been awarded and will involve major software development together with system integration.
Railtrack did not feel confident that it could reliably assess the capability of possible suppliers with respect to software development and had no way of judging whether the proposed costs and timescales were either achievable or good value for money. They turned to QSM for expert advice and followed QSM’s proven Tender Evaluation Method which supports the key management decisions from pre- Invitation to Tender (ITT) to system acceptance.
The first task was to establish a “should cost” estimate before the ITT was issued. Based on a high level description of the functionality and using QSM’s industry reference database, it was possible to set upper and lower limits on the likely cost and timescale of the development. Not only did this help internally with budgeting, but it established expectations against which tenders could be evaluated. (Suppliers often learn about their customers’ expectations and sometimes submit bids to match, even if the work can be done for less cost or even if there is little chance of meeting the targets).
Railtrack included QSM’s Software Questionnaire in the ITT and specified that tenderers must supply the required information in order to be compliant. The questionnaire is in two parts.
The first part asks tenderers to supply a few simple metrics on previous projects, preferably ones comparable to the proposal. In addition, they must supply a customer contact who can independently verify the data. The data is analyzed using SLIM or PADS to calibrate the productivity levels of the projects, which can then be compared against each other and against industry averages. “It was enlightening to see which suppliers could provide reliable data on relevant projects”, said Railtrack’s DART Project Manager, Peter Dearman.
The second part of the Software Questionnaire asks the tenderers to identify and size the software components and indicate whether they are COTS, reused, modified or to be written from scratch. It also asks for a month by month plan showing numbers of staff and major milestones. From this, SLIM can calculate the level of productivity implied in the plan.
“It was easy to see which proposals could be substantiated using the reference projects. Having a quantitative calibration of performance really helped us decide which tenderers could be relied on and which were high risk”, said Dearman. “What’s more, the QSM analysis was a potent tool for exposing inconsistencies or uncertainties in the proposals,” noted Engineering Manager Phil Clayton. “We were able to challenge parts of the plan and clarify technical and management issues, to the benefit of both Railtrack and the winning supplier” he added. The DART team were able to choose their supplier with more confidence because they understood the quantitative nature of the proposals. Now they plan to track the suppliers progress in developing the system using SLIM-Control. “We have asked QSM to present their method to a wider circle of managers within Railtrack”, said Clayton. “I know there are other developments that can bring significant benefit from the application of these methods."