DevOps Consulting Services: How Do You Measure the Work?

Nearly every DevOps consulting services contract we have been shown has a scope, a day rate and a start date. Very few contain a number that says what better would look like. Six months in, the review meeting becomes a conversation about whether things feel faster.

Five numbers settle it, and none of them is uptime or tickets closed. Two of the five you can work out from your own repository this afternoon, with nothing to buy and nobody's dashboard to take on trust.

So here is our position. Put two delivery metrics in the statement of work, measure them yourself from week one, and never let either of them decide what gets invoiced.

Why does a DevOps consulting services contract have no numbers in it?

Because it suits both sides. The vendor does not want to be held to a figure that moves with how often your product team merges anything, and the buyer does not know which figure to ask for. So the scope reads "improve CI/CD maturity", "implement industry best practices", "reduce deployment risk". None of those can be tested on a Friday afternoon.

What the first month should hand over is a different question, argued out in the post on what the first month should deliver. A deliverable is a thing that exists. A metric is a claim that the account got better, which is what the person signing the renewal is actually asking.

The first time we tried to write one into a statement of work we chose availability, and it was useless. The platform was already at four nines. The real problem was that a two-line config change took nine days to reach production, and the contract never measured that.

Which numbers actually say the delivery got better?

DORA's, and there are five of them now rather than four. The definitions published on dora.dev split them into throughput and instability.

• Change lead time: "the amount of time it takes for a change to go from committed to version control to deployed in production".
• Deployment frequency: "the number of deployments over a given period or the time between deployments".
• Failed deployment recovery time: "the time it takes to recover from a deployment that fails and requires immediate intervention".
• Change fail rate: the share of deployments needing immediate intervention, "likely resulting in a rollback of the changes or a 'hotfix'".
• Deployment rework rate: "the ratio of deployments that are unplanned but happen as a result of an incident in production".

That fifth one is recent. DORA's 2025 year in review says it plainly: "we have also officially evolved the four software delivery performance metrics into five metrics." A partner still pitching the four keys is on the older model, which is a reason to agree the count rather than walk away.

Two of the five are cheap: deployment frequency and change lead time fall out of data your repository already holds. The other three need you to agree what counts as an incident, which is a conversation, not a query.

What is wrong with making deployment frequency a target?

It gets gamed, and DORA says so on the same page: "setting metrics as a goal ... increases the likelihood that teams will try to game the metrics." Pay a partner against deployment frequency and one change becomes six deploys. The chart improves. Nothing else does.

There is a second trap in comparison. Those metrics "are meant to be applied at the application or service level", and comparing them between very different applications "can be misleading". Your settlement service and your marketing site were never on the same scale. Measure each against its own last quarter.

Why you want both halves together is in the 2025 DORA research. Google Cloud's announcement of that report, drawing on "survey responses from nearly 5,000 technology professionals from around the world", found a positive relationship between AI adoption and delivery throughput while noting that "AI adoption does continue to have a negative relationship with software delivery stability". Speed is the easy half to deliver and the easy half to put in a slide, so a report carrying only a throughput number is showing you the pleasant half of the result.

Which metrics will a partner report instead?

Four come up again and again, and each is weaker than it looks.

• Uptime. Mostly a property of the architecture and the traffic, not of this quarter's work. A platform that was stable in January is still stable in June.
• Tickets closed. A count of what was asked for. A partner who closed 140 tickets and one who removed the cause of 140 tickets look identical here.
• A maturity score out of five. Useful once, at the start, to settle the order of the work. As a monthly trend it is the supplier marking its own homework against a scale it wrote.
• Infrastructure-as-code coverage. Worth knowing, and load-bearing for an audit. It says nothing about whether a change reaches production faster or more safely than it did in March.

Where we concede ground: the opening assessment earns its fee when it changes the order of the work, and coverage is the one of the four worth reporting monthly. Neither replaces a delivery metric.

Does the RBI ask for any of this?

More directly than most teams expect. The RBI Cybersecurity Directions 2026 for NBFCs say at paragraph 148 that the entity "shall implement suitable scorecard / metrics / methodology to measure IT performance, and IT maturity level". Paragraph 41 wants a board-approved change management policy covering "change implementation, monitoring, and reporting". Paragraph 62 requires the vendor contract itself to carry "provisions for continuous monitoring and assessment of the service provider".

Nothing in that text names DORA, and nobody should pretend it does. What it asks for is a scorecard that exists, gets reviewed, and covers the supplier as well as the platform. A monthly delivery metric with the deployment list attached answers paragraph 148 for the delivery half of IT performance. A maturity rating written by the supplier does not.

Keep paragraph 147 separate. It asks for "suitable metrics for system performance, recovery, and business resumption, including RPO and RTO", which is resilience measurement and a different exercise. The rest of the direction is in the post on the 2026 direction, and what paragraph 62 means for the contract wording is in the outsourcing post.

How do you measure two of the five on Monday?

Out of the GitHub Deployments API, if your pipeline creates a deployment record. GitHub's reference for GET /repos/{owner}/{repo}/deployments takes an environment filter and returns each deployment's sha and created_at. That is deployment frequency, and lead time once each sha has its commit date.

ORG=your-org REPO=your-repo ENV=production

# Deploys per day, newest first
gh api --paginate "repos/$ORG/$REPO/deployments?environment=$ENV&per_page=100" \
  --jq '.[].created_at' | cut -c1-10 | sort -r | uniq -c | head -30

# Hours from commit to deploy
gh api --paginate "repos/$ORG/$REPO/deployments?environment=$ENV&per_page=100" \
  --jq '.[] | [.sha, .created_at] | @tsv' | head -50 |
while IFS=$'\t' read -r sha deployed; do
  committed=$(gh api "repos/$ORG/$REPO/commits/$sha" --jq '.commit.committer.date')
  echo "$sha $(( ( $(date -d "$deployed" +%s) - $(date -d "$committed" +%s) ) / 3600 ))h"
done

Two things caught us. The second loop needs GNU date, so on a Mac use gdate or a container. And lead time from the merge commit is not the same number as from the first commit on the branch. Pick one, write it down, and do not change it mid-engagement.

The instability half needs a convention before it needs code. Label the pull request that fixes a bad deploy hotfix, and the issue recording a production incident incident. Change fail rate is then hotfixes over deployments, rework rate is incident-driven deploys over deployments.

What should the statement of work actually say?

Four lines, in your words rather than the supplier's.

1. Two named metrics, one throughput and one instability, defined by reference to the published DORA definitions rather than restated in the vendor's prose. Change lead time and change fail rate are the usual pair.
2. A baseline measured in week one, by you, in your own tooling, before any work starts.
3. A direction of travel rather than a figure: lead time falls, fail rate does not rise. A target number invites the gaming DORA warns about.
4. Monthly reporting with the deployment list attached, and no payment trigger on either metric.

All of that assumes the repository and the AWS account are already in your name, which is the subject of what you should own when a consultant leaves. If the deployment records live in the vendor's GitHub organisation, you do not have a baseline. You have their word for one.

When is measuring nothing the honest answer?

When there is nothing yet to measure. If production changes because one engineer with console access makes it change, the first month's numbers are noise and the deliverable list is the contract. Measure from the month the pipeline exists.

Below about five engineers, one person reading the deploy log every Friday beats any scorecard, and the comparison between hiring that person and bringing in a team is in the post on hiring a DevOps engineer or a partner.

A partner who declines a number in month one but commits to one from month three is being straight with you. The one to worry about accepts whatever metric you propose, immediately, without asking how it will be calculated.

The short version

Pick two of DORA's five, one throughput and one instability, by the published definitions rather than the vendor's. Baseline them yourself in week one, before the engagement touches anything, then report a direction of travel every month with the deployment list behind it. No target figure, no payment trigger, no comparison against another team's service.

Agree the hotfix and incident labels this week. And if you are regulated, remember the scorecard is paragraph 148's problem too, which a supplier's own maturity rating has never answered.

This is engineering guidance, not legal advice. Regulations and their annexures are amended over time; read the current text and take your compliance position from counsel.