You are neither right nor wrong because the crowd disagrees with you. You are right because your data and reasoning are right. Warren Buffett
Recently one of our customer asked for help in building a business case for their ALM transformation project. The customer wanted to prove to their stakeholders that the required investment in ALM was worth it.
We started by providing some “out of the box” statistics but we also asked them why and what they wanted to achieve, bringing their own context into the equation.
At first the customer thought that using TFS to gather all the data, and turn it into information, would suffice. However, going through the process it became clear it would not. The end result was far from what they originally imagined, however it was aligned with what their goal was, i.e. prove the value of investing in ALM being done right. So what did we do to get there?
Before we start, remember that all models are wrong but some are useful (George E. P. Box), there is no cookie-cutter approach to measurement nor there is a silver-bullet to solve challenges.
We decided to use the Goal Question Metric (GQM) method/approach to help the customer define its goals and metrics. GQM is an approach to software metrics developped by Basili and Weiss (1984), then Solingen and Berghout (1999) that defines a measurement model on three levels:
Other approachs like the Value Focused Thinking (VFT - Keeney, 1992, Keeney, 1996) approach and The project objectives measurement model (POMM - Barclay, C. and Osei-Bryson, K-M 2008), could have been used. Even if GQM is intended for software organizations, the focus was more project/process management than true software measurement (e.g. lines of code) and we had experience with it.
First, we started by defining the goals, why they wanted to improve, why the investment was needed. Second, based on those goals we defined the questions they wanted to answer, what needed to be looked at. Third, we defined the metrics they needed to measure to help answer the questions and ultimately meet their goals. For example:
Goal:
Questions:
Metrics:
Keeping in mind that the software development methodology and process had an impact, i.e. the metrics could be different for an Agile project vs. a waterfall project.
And ended up with something like this:
We were not done though. We realized that we had many metrics that needed to be prioritized, so we looked at low hanging fruits and importance/impact on the business, weighting them to get to priorities. We then looked at the “balance”, for example:
- We then looked at how to get the data and from where/which system, it became clear that TFS was not the only answer.
- We re-prioritized again based on the how, being careful not to only include what was measurable (easy to do) but more importantly what was meaningful.
- We looked at how to report the data, what had to be represented in graph, what kind of graph, timeline and intervals to collect data, showing trends or detailed values, consolidating on a dashboard, etc.
- We left out targets for the first phase, we first needed to collect historical information before we could set meaningful targets, regardless if we decide to do it per team, per process, per applications, etc. and how we decide to do it (thresholds, peer-pressure, etc.).
- We also defined the implementation plan: change in process, training required, systems to be customized, etc.
- We decided to use GQM but don’t be married to a model.
- Like my former esteemed colleague used to say “it’s like apple and tractors, they are both red/green but they don’t do the same thing”, make sure to compare apples to apples, for example if you measure “bugs” how you manage them can be different in Agile and in Waterfall.
- Tools are not silver bullets, don’t confuse measurable with meaningful.
- GQM is easy to understand but not that easy to put in practice, you can get off-track easily, jumping to metrics directly without defining and challenging the goals. Make sure to always ask yourself why and get a third-party to assist so you get a pair of fresh eyes.
- You can download this spreadsheet that provides GQM examples and a list of actions: Goals-Questions-Metrics example
Finally, “You got what you measure, measure the wrong thing and you get the wrong behaviors.”
― John H. Lingle