Updated: May 9, 2019
APRIL 24, 2019 BY JAKE BREWER
Think back to a substantial work project. Most likely, it didn’t just start with large and expensive actions like development. Instead of making big decisions without considering the consequences, there was probably an evaluation of the situation and some initial research. There were most likely S.M.A.R.T (Specific, Measurable, Achievable, Relevant, and Time-based) goals. This article will focus on the M of SMART. It is difficult to improve what is not measured.
When looking at measurement as it pertains to IT, specifically technical debt. Measurements are an effective way to prioritize debt and decide how to proceed. After all, the level of success of any business change cannot be determined without knowing the measurements before and after the project.
It’s also important to note that measuring the size and flavor of technical debt itself is an important way of communicating the concept to non-technical team members and stakeholders. Measurements can express the estimated number of hours required to address technical debt. Measurements can help quantify and validate the results of debt reduction efforts. Not only do measurements relay these essential statistics, when presented well they can provide greater understanding of change to the laymen on the team.
What to measure?
Lay the groundwork for measurements by determining Key Performance Indicators (KPI) that are to be targeted. Knowing the definition of success transforms an idea from an abstract concept to an attainable goal.
In any situation, it’s important to measure the resources going into a product or process; this can include hours, dollars or hardware. Determining the cost of these resources upfront will enable understanding of the impact caused by changes.
It is important to choose what to measure and with whom to give visibility with great care. Measurements drive rewards. Rewards drive actions. Rewards are a change, and their impact should also be measured.
For example; let’s say a team is trying to improve code quality. They choose to measure the number of escaped defects in a release. This may be the right measurement, but if they use only that measure; they will likely have unintended consequences. Possibly in the form of longer cycle-time and fewer features making it out to the release. Probably, not the intent to forfeit speed to market for quality. Without the proper measurements and an expert guide to help the team along the way, a team can end up just trading one problem for another. Rewards drive actions but, not always the actions intended.
Another example; Teams trying to measure software development often decide to use Lines Of Code (LOC) as a metric. While this metric has uses, when used alone this metric drives some destructive behavior. Like any measurement rewards may form around LOC. Developers that write bloated code or excessive comments are rewarded, others see this and begin to do the same.
Here are some factors that should be considered for measurement:
Quality loss: Gain a big picture overview of quality losses in production through the measurement of defects. This can include number of defects, business losses, financial cost, and reputation damage caused by any defect.Parameter impacts: Optimize parameters (like a team’s target unit test coverage) by measuring their impacts on efforts (quality loss?). This will help quantify the effectiveness of efforts. Trends over time: Consistently and persistently measure trends in product and process. Record and measure any deviations in the process as well. This will not only demonstrate the importance of the process and meeting the highest technical standards but give additional clarity into the cost/benefit of the deviations.Code test coverage: What percentage of the code is covered by unit tests. Code security: What security vulnerabilities are discovered? Cost to fix?Test quality: Identify tests covering any area where a defect was found. Code quality: When and how did any defect get created. Defect remediation: What was done in response to a defect
When to measure?
Measurement should be done before, during, and after any change.
Before a change is made, measurements should be put in place and left to collect data for a significant duration of time before making any changes. Without this it is impossible to know if anything is improving or degrading. We call the setup and collection of initial measurements initiation.
During a change, the measurements will give an ongoing idea of how things are progressing. Are the KPI measurements improving? Are improvements on the KPI translating to the business benefits expected?
After a change quality measurements will give a thorough understanding of the impact.
How to measure?
Measurements should be as automated as possible. Although sometimes unavoidable, relying on manual measurements is prone to subjectivity, forgetfulness and other human shortcomings.
Fortunately, many specialized software on the market can help do this automatically.
A CI/CD pipeline – report on several well known KPI for software development success.Project management software – track rate of progress.Bug tracking software – track deliverable quality.Log monitoring tools – aggregate system logs.
Areas to consider when looking for KPI:
At IT Efficiency Consulting, we fully believe in the importance of measurements before, during, and after the optimization process. By putting in the time and resources at this measurement phase we: not only prove our value but, more effectively use our resources and our client’s resources. If you are interested in improving company productivity, turnover, and quality through lowering waste in the area of technical debt the right way, send us a message. We’d love to hear from you and help you get started on the path to higher quality software, lower turnover, and resource savings today.