Metrics That Matter: Getting the Full Picture on Digital Delivery
This article proposes taking a holistic approach using both product-related and performance-related metrics and putting information provided in context.
As the old adage goes, if you can’t measure something, you can’t manage it – and you certainly can’t hope to improve it. So it’s no surprise that, as companies accelerate down the road of digital transformation, many are looking for new and better ways to measure their progress.
The trouble is that traditional ways of measuring the ability of development teams to deliver software don’t give the full picture of what’s happening. And that, in turn, makes it difficult for managers to make good decisions when it comes to prioritising work items and allocating resources to them.
That’s not to say that delivery metrics aren’t useful. Throughput, for example, is a valuable measure of the rate at which new code is put into production. Lead time gives helpful insight into the time that elapses between the identification of a new requirement and its delivery into production. Cycle time, another beneficial metric, indicates how much time elapses between the actual initiation of work on a piece of code and its go-live.
Throughput vs Lead Time.
But the reality is that, in many organisations, and especially larger ones, these metrics can be highly unstable, especially if managers resort to throwing more bodies at a problem in the face of an impending deadline.
More importantly, they don’t tell the full story. Other dimensions are needed in order to get an accurate measure of progress against the digital transformation roadmap. In particular, what’s missing from the picture are product-related metrics: the quality of code, for example, as revealed by defect rates; and whether the overall level of technical debt (the amount of later rework incurred when the delivery of a piece of code is expedited) is either mounting up or being whittled away over time.
Throughput vs Technical Debt vs Defects
These product-related metrics matter, because of their close interplay with the performance-related metrics we discussed earlier. Pushing harder on throughput, for example, can cause defects or technical debt to rise. Conversely, work to reduce technical debt may lead to a corresponding dip in throughput. Only by collecting this data and combining it in order to identify emerging patterns can management understand the full implications of their decisions on overall progress.
At Zuhlke, we are taking a metrics-driven approach that seeks to be more holistic by focusing on both types of measure. We find it useful not only in our own work, but also when we assess the performance of our clients’ digital delivery teams and make our suggestions as to where and how they might improve in order to achieve their digital goals. Our approach focuses on introducing product-related data drawn from JIRA ticketing systems and GitHub, for example, and combining it with performance-related metrics in easy-to-interpret dashboards.
These dashboards provide an at-a-glance way for users to identify patterns and trends – and from there, with the right training and knowledge of business context, to find correlations between decisions made and upticks/downturns in performance.
The information provided by the metrics needs to be interpreted in context—the measurements in themselves cannot indicate if a situation is good or bad. For example, an ‘all hands on deck’ diversion of resources in order to get a big new customer on board may have a negative impact on technical debt, lead time, and cycle time. However, if the value provided by the acquisition is much higher than the cost of fixing the problems caused by it, the diversion was a good decision.
Metrics, usually, are lagging indicators, but there are some that can be used almost in a predictive manner. For example, combining the measurement of technical debt with frequency of change of source files, may help to decide where a concerted effort on reducing technical debt is likely to give the biggest long term gains in terms of throughput, cycle time, lead time, etc. Done well, this kind of analysis leads to insights that the organisation can take forwards into future projects.
Either way, training is essential. A management team that is simply drowning in data, with no way to make use of it, is likely to lose its way pretty quickly. But one that knows how to extrapolate visual clues, interpret them correctly and use their findings to make more informed decisions will immediately find itself on a much firmer footing. And, with the metrics that matter under its belt, we believe any organisation can take on the digital transformation challenge with renewed confidence.