I have spent the past couple of years as the owner of some scrum teams and when I integrate that experience with years of doing, teaching, and coaching scrum I find that I have finally found some good ways to use metrics. This article is for all of you managers and scrum masters out there who might be looking for some suggestions.
Here are some things metrics won’t do for you:
• Metrics won’t tell you if you people are “working as hard as they should”
• Metrics won’t tell you which individuals are “good” or “better than others”
What I’m saying is that just because you have metrics doesn’t mean you should use them any way you feel like. Used incorrectly, they will sap the will of your teams and employees and eliminate the gains you might otherwise get by using scrum. The funny part is that there isn’t any metric out there that will tell you if your teams have become uncaring, unmotivated, disengaged, unproductive drones due to your misuse of metrics!
So let me first dispense some warnings. Here’s what not to do with metrics:
• Don’t use team metrics to compare the “performance” of Scrum teams
• Don’t use team metrics to either reward or punish individuals regarding pay or promotions
• Don’t use team metrics to make customer commitments
• Don’t use individual metrics, period.
• Don’t try to measure “software product development productivity.”
“Sheesh.” I hear you cry. “It seems like there’s nothing I can do with metrics, then.” Fear not.
• Do use metrics as part of an inspect and adapt process to improve team performance
• Do use metrics to gauge whether a team has a healthy or unhealthy agile development process in place
• Do encourage, allow, and help your teams to use metrics to improve.
• Do rotate your use of metrics. Don’t use the same one for more than six months.
So what metrics can I use to tell if my teams are doing well? Here are a few:
• Total Defects
• Total Defects in Agile Code (reported after story completion, aka “escapees”)
• New Defects in Agile Code
• Defects in Legacy Code
Total Defects tells you something about the condition of your entire product. Total Defects in Agile Code and New Defects in Agile Code tell you how well you are doing in achieving the Agile potential of zero defect code. You should be looking for total defects to level off over time, and then to turn south and head towards zero. What you do to make that happen is a subject for another day, but you can’t attack defects it until you can see your defect picture on a nice, simple graph.
Does it go without saying that if you are doing Scrum and you can’t get these under control then you have some significant problems in either your scrum process or your engineering practices? I hope so. If it doesn’t, please hire me as your coach.
• Do/Say Ratio in Points
• Do/Say Ratio in Stories
• Number of stories not completed per sprint
The Do/Say ratio is a way to capture the team’s ability to commit (or predict, depending on which school you are from) and then deliver predictably. Nobody can do this perfectly but a good scrum team can do it very very well. If these metrics aren’t near 100%, then you should look for inconsistent estimation, inappropriate story size, organizational abuse, or persistent waterfall engineering practices.
The ratio is expressed as (what you actually did, or “Do”)/(what you thought you’d do, or “Say”). 100% means you did exactly what you said you would do. 80% means you did less than you expected, and 120% means your team outdid itself. Not being able to predict means that the team doesn’t have a good handle on either the work it is doing or how it is being done.
I find that I want to track both the percentage in points and also I want to know how often a story is missed. Again, this is a general health indicator and not some kind of punishment/reward thing. For instance, if I see that a story was missed twice in the last three or four sprints, then I can ask the Scrum Master if the team has retrospected on this troubling occurrence.
Danger: If this metric is misused, the team will find a way to make these go up by predicting that it will do much less, thereby making it easier to hit 100%.
• Average cost per Story Point (in $)
• Average cost per Story (in $)
These take a bit more thinking, but they can be nice to have around. Cost per story point is exactly what it sounds like. It is the total cost of the team per sprint (fully loaded salaries times the length of the sprint probably) divided by the number of points of work accepted. It’s awfully fun to be able to give a quick answer to the traditional “We have a customer who wants such-and-such. How much will it cost to build?”
Multiply the cost per story point times the average size of a story in your Product Backlog and you get Average Cost per Story. Are these interesting? Heck yes. Sometimes the ACPSP goes down (meaning the same team is doing a lot more points per sprint) but the ACPS stays the same (meaning the team is inflating estimates?).
And finally, here’s one that is very uninteresting:
Velocity is only good for one thing, which is to predict progress down an estimated Product Backlog. It’s highly volatile and it doesn’t travel well. Not only is it specific to a single team, it tends to change over time.
OK. There are a few metrics you can use with a scrum team. There are more out there just waiting to be invented and used. Make sure you use them for good and not evil.