“The fog of war” - What we don’t know does not hurt us - Carl von Clausewitz
Of course cost of software development is relatively easy to calculate right? Many studies, models and methods exist to estimate the cost of the development organization; things like number of lines of code (LOC), K-LOC, function points, weighted micro function points, price systems, wideband Delphi paired with cost of resources, COSMIC, analogy method, COCOMO or even metrics linked to project management (EVR, BCWS, ACWB, BCWP, etc.) and fully loaded cost of resources (cost of training, the % utilization (billability) of resources, etc.).
But assuming you are an agile organization, theoretically cost measurement can be done with concepts like PBI/user story points, which can be turned into dollars - cost per point (4), if you can measure the cost of an individual, or a group of individuals (the latter is preferred).
Right?
Well, Buffett found it 'extraordinary' that academics studied what was measurable, rather than what was meaningful, following is why I don’t think measuring some of the things above is meaningful.
Recap of a recent discussion at a customer (Names have been changed, they are following a Scrum process):
“Hey Joe, how are you? Is the sprint going well?”
“I’m fine Frederic, thanks. Sprint is going well. It’s obviously not easy, as you know we lost a resource and are lacking some technical skills but we are working hard to deliver by the end of the sprint”
“Great Joe! How are you compensating for those missing skills? Training, coaching? Are you looking at getting an experienced resource on the team?”
“We are looking at adding a resource yes, but right now we are learning those skills on our own, with some coaching from someone from another team. She helps when she has the time.”
“So did you remove some work from the sprint to make it fit? Or did you fail the sprint?”
“No, the PO did not let us change the sprint backlog so we are working hard to learn. Team members are working week-ends and evenings to make up for the time.”
…
As you know in Agile the time spent is not as important as time left, and as we commit to deliver as a team, what if team members don’t log all their hours? Wouldn’t the cost calculation be skewed?
And as the team does not have the skills, wouldn’t the LOC/KLOC calculation could be misleading? Their code is more than likely not as optimized as one from an experienced resource.
…
“Did you discuss those challenges with the team during the retrospective?”
“Yes, of course, we are looking at implementing some changes.”
“Ok, any other challenges discussed during the retrospective?”
“In fact yes, and maybe you can help actually, we are still struggling with estimation. Even after 6 sprints, our user story points and our estimation of tasks are still inaccurate. Any suggestions?”
…
So turning PBI/user story points into dollars (cost per point), will be inaccurate as well.
Those are just a few examples (5) of what can impact, negatively, the measurement process. What does it mean? It implies that:
- Metrics should be defined in the context of the customer, indeed, even if the overall goals are the same (reduce cost, improve quality, increase productivity, etc.), the metrics could end up being different based on what is meaningful for, important to, the customer; and
- The analysis of the results, should be put in perspective as well, and that does not require a complex measurement system, but it does require an understanding of the environment, which can rarely, or not easily anyway, be automated (yet?). You have to observe and analyze. (6)
The good news is that as we understand the relationship between improving productivity and quality improvement, we know that implementing concepts like inspection of code, unit testing, test driven development, daily stand-up for tracking progress and small self-managed teams, close to the customer (or PO), are beneficial… (and some coaching for the PO in my example!). Also metrics like code maintainability, complexity, etc. should be looked at. Implementing those concepts will more than likely improve software development (even if we cannot precisely quantify and qualify (7)). The ALM Microsoft platform is great for that! Many of those features are in Visual Studio.
But at the end, cost should still be measured in dollars (or any other currency). I believe the cost (fully loaded) of a ‘point’ should be a good start if you want to measure the cost of software development.
-------
- The scope of this blog post is software development only, and around cost and value only, there are many other metrics to be considered, like customer satisfaction, ratio of maintenance vs. new development, cycle time or time to delivery to name only a few. Measuring performance should be done using an aggregation of metrics, not only cost and value.
- Theory of constraints (Derick Bailey).
- This example is to illustrate my point, not to take a political stand.
- Dave Thomas: “Story points are training wheels for people that grow up to do real estimates”. That one really got people going!
- Obviously that organization has many other challenges!
- HINT: Keep in mind the Mayo experiment at Hawthorn, just putting in place a measurement plan will improve the organization. Therefore instead of putting targets in place for metrics, we might be better off with analyzing results and comparing those results with the historical data, answering the question: are we getting better. Also as per the previous blog entry on gamification, we should make the results visible, transparent and introduce the concept of positive “peer pressure” (Team 1 compare to team 2). Just make sure people understand why you are doing it and don’t perceive it as personal evaluations.
- Isn’t that common sense? I’m not suggesting to only do Agile (or/and Scrum), I believe you should follow a process that works for you. Keeping in mind, to quote my good friend Alain, that “You can produce diving suits made out of concrete by following an ISO process”.
