Change is never easy. Change is even harder when you’re unsure whether your DevOps implementation is changing your team/application/organization for the better or worse. One of the biggest mistakes organizations make when adopting sweeping process or technology changes is a failing to identify measures to determine whether they are trending in a positive direction and when they have been successful. It’s important to identify the objectives of the project and how those objectives will translate into the appropriate metrics for tracking progress. Simply “feeling like it’s better” is not enough.
Over time and through my experience, I have found that there are five essential quantitative metrics for any DevOps transformation.
1. Deployment Frequency
What it means: How often you deploy a new version of a specific product or service as measured over a particular period of time (day, week, month, year).
Why it matters: Frequent deployments suggest there is continuous development, testing, and integration of small changes that will continuously improve an application. Also, frequent updates suggest that the organization can be responsive to business requests for new features and rapidly respond to critical issues.
Metric Expectations: This metric should trend up or remain stable from week to week.
2. Deployment Speed
What it means: How fast you can deploy a new version of a specific product or service to a particular environment.
Why it matters: Traditionally, systems were highly coupled and involved complex architectural dependencies that forced full system deployments every time there was even a minor update. While that may have been acceptable years ago, today’s enterprises have a customer base that’s always awake somewhere, so minimizing downtime is critical. Increased speed of deployment reflects an increased consistency from commit to production, with automation removing manual, error-prone manual work from the DevOps pipeline.
Metric Expectations: This metric should trend down or remain stable from week to week.
3. Lead Time (from development to deployment)
What it means: How fast code takes to make it to production from the point you start development.
Why it matters: Lead time can help us determine how fast we can respond to changing user needs. Projects that have not successfully adopted agile will often have large lead times. Tracking lead time and the subsequent components of the process can help the team determine areas in need of improvement.
Metric Expectations: This metric should trend down or remain stable from week to week.
4. Failure Rate
What it means: How often we fail over a period of time; this can measure production failures as well as failures in our testing, deployment or DevOps pipeline.
Why it matters: It’s great to deploy more frequently and quickly, but if changes fail just as frequently, you’ve gained nothing. Failed deployments can take services down, resulting in lost revenue and frustrated customers. Well-implemented DevOps practices can make a big difference in failure rate.
Metric Expectations: This metric should trend down or remain stable from week to week.
5. Mean Time to Recover
What it means: How long it takes the application to recover from a failure, as an average over time.
Why it matters: When failure does occur, how long does it take the team to recover from the issue? This is a strong indicator of how good the team can handle change and respond to issues. Failures are inevitable, but how we respond is the far more important metric of how agile our team is.
Metric Expectations: This metric should trend down or remain stable from week to week.
