Businesses are always looking for ways to increase efficiency and eliminate waste. For software-driven companies, DevOps is one approach that helps to achieve this. The goal of DevOps is faster software delivery to the end-user while maintaining high software quality. DevOps enhances collaboration between operations and development teams for faster code deployment.
When evaluating the effectiveness of your DevOps model, it is critical to use metrics relevant to your organization. The best approach for measuring success is to identify the key outcomes you want to achieve, and then find the right DevOps metrics to monitor those outcomes.
For example, you may want to deploy new functionality faster or shrink the recovery time when things go wrong. The actual metrics used for these outcomes will differ from organization to organization. If the goal is to deploy new functionality faster, one organization could measure deployment time while the other measures lead time.
Below are eleven DevOps performance metrics you can track to gauge the success of your DevOps approach.
The entire point of DevOps is to get updates to your customer as quickly as possible and with the highest quality standard. This is why it is critical to measure the frequency of deployment. The more often you can deploy new code in the production environment and make it available to customers, the better.
The time from commit to deploy shows how effectively you can make new functionality available to customers. As soon as an engineer has completed a feature, you want it in the hands of users delivering value. Functionality shouldn’t be sitting in queues for review or otherwise held up - any delay between code-complete and production is waste. Shortening deployment time reduces that waste.
When an engineer checks in new code, how long does it take before it is integrated into the source code system? Longer pull request cycle times may be indicative of an ineffective DevOps process.
Lead time measures the amount of time it takes to implement, test, and deliver code. In general, it is the time that elapses between starting a work item until the time it is deployed. As with other metrics measured by DevOps, you want to keep your lead times short and fairly stable over a time period.
5. Change Failure Rate
As developers bring changes into production regularly, a certain percentage of those will have problems. That’s natural, and not a big problem as long as you keep a short MTTD and MTTR (see below). But it’s important to measure the ratio of changes that pass to changes that fail. Once you establish a baseline, make sure that the failure rate doesn’t increase significantly - that could point to problems in your DevOps pipeline.
The raw count of errors reported by your system when changes are introduced in production is a good metric for DevOps. A smaller number of errors indicates high-quality functionality.
The mean time to detect measures the average time it takes to discover a problem in production. How long does it take to detect an issue after changes are introduced? And how long does it take to get notification about that issue to the people who can remediate it? Measure that time and make sure it's as small as possible to give you time to react when something goes wrong.
When reporting on production issues, it’s important to categorize them by severity. For example, a high-severity issue might be one that has a major impact on core functionality for all your customers, while a low-severity issue might have just a minor impact or might impact only a subset of customers. You should be concerned if most of your issues are high severity, even if their number is low. Whereas you may be comfortable living with a higher number of lower severity issues if the payoff is much better velocity.
Having failures is inevitable, and trying to eliminate all possible issues before releasing software is a guarantee of low velocity. What’s important is measuring how long it takes to recover when something inevitably does go wrong. Once an issue is known or detected, what's the mean time to repair? That is, the time it takes to either remove that change from production or fix it.
While the mean time to repair measures the length of time it takes to repair the system (usually up to the testing phase), the time to recover measures the entire time it takes from when the system fails to when it becomes fully operational again. How much time does it take from when an issue is identified to when it is completely resolved.
While building stable, high-quality products is important for success, it’s equally as important that those products are economically viable. Unit cost is a great way to track how costs are trending in the context of your business.
This varies based on your company, but can include cost per customer, transaction, video stream — or whatever other metrics meaningfully align to your business model. It’s also important to understand which parts of your architecture are driving those costs.
If the goal of DevOps is to maximize efficiency and eliminate waste, then an important part of gauging its success is to monitor cost alongside other DevOps metrics. In a perfect world, everyone would have the most scalable, fast, and highly available applications possible.
However, companies also need to ensure they can sell their product at a price the market is willing to pay while maintaining strong margins. Achieving those attributes often requires you to spend more — either in actual cost, or in time spent building and maintaining.
It’s important to consider the underlying cost implication of each DevOps metrics and decide which tradeoffs make the most sense for you. For example, if you’re a payment processing platform, ensuring high availability of your payment functionality is likely extremely important for your customer experience and trust — so you may want to accept higher costs to support that feature. For less essential products, you may want to choose to value cost-efficiency higher instead.
CloudZero makes it possible to correlate costs with important DevOps activities, which is key to maintaining control. DevOps uses a lot of different tools — continuous integration servers, source code control servers, and continuous deployment tools — behind the scenes. CloudZero programmatically interacts with all of those systems, making it possible to correlate your costs over time with the activities in your DevOps pipeline.
For instance, you might measure the frequency of code deployment as a DevOps metric. Every time a team pushes a change live in production, CloudZero records that, so you can see the impact to your cost when that deployment happens. In the screenshot above, CloudZero captures a cost spike following a deployment.
This way, you can see the deployments contributing the most to your cloud spend and find ways to reduce those costs, where necessary.
By providing this level of insight and cost intelligence into your development process, CloudZero helps you not only improve your DevOps process but also to build cost-optimized software.
