Every product organization is drowning in data. Sprint velocity, story points completed, lines of code written, deployment frequency – the list goes on. Yet despite all this measurement, most product leaders still can’t answer the fundamental question: “Are we actually getting better at building and shipping products?”
The problem isn’t a lack of metrics. It’s that we’re measuring the wrong things. Or we’re measuring so many things that the signal gets lost in the noise. Or worse, when metrics become mandates from above, teams start gaming the system rather than using metrics to inform smart decisions to improve their performance.
Metrics change behavior. If you as an organization measure how many deployments you can do, then developers might push out 10 deployments a day because it makes their numbers look really good. But it could be of little value – users might not want that many changes in a day, or there might not be value in having that many changes.
This is the metrics paradox: measure everything, improve nothing. But there’s a better way.
The four questions every product team should ask
Rather than getting seduced by vanity metrics, successful product organizations focus on four fundamental questions that reveal the health of their development process.
Over the last 2 years, we worked with a client on these very questions. Firstly, we needed to understand the current state. The next step was for the team to agree on a continuous improvement path. Then, over time, the team experimented and measured to gauge whether the change had a positive impact. Ultimately, over the 2 years the team was able to improve backlog health, delivery commitments, velocity variability, and bug rate.
Here are the four questions we focused on:
How good are my inputs?
This speaks to the quality of requirements/user stories and the health of the product backlog. Is the product team feeding high-quality, well-understood, customer-informed work into the development pipeline? Low quality inputs lead to wasted time, rework, and inaccurate estimates. Measuring input quality helps you identify upstream issues and ensures the team is working on the right things at the right time.
Metrics that could help you measure the quality of your inputs include:
- Requirement Completeness: % of user stories that have acceptance criteria, designs, and business context defined before development begins.
- Change Rate: How often requirements change after development has started.
- Backlog Churn: % of items that are added/removed/re-prioritized within a sprint cycle.
- Cycle Time to “Ready”: Time from initial idea → grooming/refinement → ready-for-dev.
How good are my estimates?
The quality of your estimates determines the predictability of your entire product roadmap. If your estimates are consistently wrong, it’s impossible to make reliable commitments to customers or stakeholders. Evaluating the quality of your estimates helps you understand the accuracy of your planning and identify if there are underlying issues in your processes, like a lack of information or unexpected technical debt.
Some metrics to help assess this question:
- Estimate Variance: Estimated story points vs actual.
- Velocity Variance: Story points completed per sprint.
- Sprint Predictability: % of committed work completed in a sprint (a healthy range is usually 80–90%).
- Forecast Reliability: Accuracy of multi-sprint or quarterly delivery forecasts based on estimates.
How good is the quality?
This question assesses the quality of the work and the efficiency of the team. Focusing on quality ensures the team is not just busy but is building the right thing. Measuring for quality ensures that the work produced is reliable, maintainable, and meets the users needs.
Metrics that can help assess quality:
- Defect Escape Rate: Bugs found in production vs. caught in testing.
- Mean Time Between Failures: Frequency of incidents, outages, or significant issues.
- Rework Rate: % of development time spent fixing/refactoring vs. delivering new value.
- Customer Satisfaction: CSAT/NPS or in-product satisfaction scores.
The key here is to focus on the quality of output, as this impacts everything that happens downstream: customer trust, engineering efficiency, and business outcomes.
How good is my pipeline?
Typically, this is where most teams spend their energy for improvement. Google publishes an annual report around this called DevOps Research and Assessment (DORA). The metrics used to generate the report are a good measurement of the effectiveness of your build and deployment pipelines but it doesn’t provide much insight into issues with input or estimation.
The DORA metrics include four key indicators of software delivery performance:
- Deployment Frequency: High-quality work enables frequent, stable releases.
- Lead Time for Changes: Time from idea → production; shorter and stable often correlates with higher quality processes.
- Change Failure Rate: % of deployments causing incidents, outages, or significant issues.
- Mean Time to Recover: How quickly the team resolves issues in production.
How do you choose what to measure?
Here’s where many product organizations go wrong: they assume more measurement equals better insights.
I think of it like golf – a game I’ve been playing for a while. Though I’m still not very good at it, I keep working to improve. Golf has lots of measurements you can take, and a lot of different aspects of your game that you can try to improve to become a better golfer overall, either while you’re playing a round of golf or practicing. In my journey to improve I started with the basic metrics: the score, number of fairways hit, number of greens in regulation, and number of putts. Then I wanted to see if I had a pattern in my misses: am I missing left or right or short or long or is it truly random? These metrics helped me identify where I needed improvement to become more consistent.
So what did I do? I went to YouTube of course. There are countless YouTube golf coaches that have the ‘fix’. I spent a lot of time going down that rabbit hole and finally decided to get a few lessons with a local coach. They were able to help assess my swing and more importantly help to prioritize changes to my swing and my game strategy. I went from struggling to break 90 to consistently shooting in the 80’s. It might not sound like much but it really is much more enjoyable to play knowing I have a general idea on where the ball is going and how to correct the issue if I see a pattern of misses. My journey for improvement continues, now that I’m able to get close to the green in 2 or 3 shots, I need to focus on improving my short game to move to the next stage in improving my scoring.
In addition to the basic metrics I mentioned, there are lots of tools to help you measure things in golf now. They have sensors on golf clubs, lots of different apps and technologies. But they all take away from focusing on the game at some point, because you’re having to press buttons and do all these extra things that don’t necessarily go into playing the game.
That’s the tax of metrics: which metrics are right for the organization based on where you are in your maturity/proficiency? And how much of a metrics burden do you really want to collect, analyze, and report, given that it’s extra effort for the team?
It’s always a fine balance. You can measure everything, but how much gain are you going to get out of that? You can measure nothing, but how can you tell if you’re improving?
From measurement to improvement
The client I mentioned earlier in the article followed a path to improvement that ended up taking two years. During the two years, metrics show they improved in commitment delivery, velocity consistency, and escaped defects.
To improve in all of those areas, they developed shared scrum training across the organization, focused on improving story quality, steadily made improvements in the estimation process, and made large improvements in pipeline automation, especially with automated functional tests as well as non-functional tests.
As the client made improvements, they kept challenging themselves to make incremental progress. Going back to the golf analogy, they went from a mid-teen handicapper to a single digit handicapper. But they had realistic expectations on the investment needed to make the incremental progress (e.g. they didn’t have delusions of going from a mid-teen handicapper to being a scratch golfer overnight).
The metrics you measure should evolve over time
Your metrics should not be set in stone nor should the targets. These are things that should help you figure out where you want to improve on a continuous basis. You don’t want to be changing things too frequently, but you also don’t want to be optimizing for metrics that are past their use-by date.
This natural evolution happens when you keep focusing on the right things to measure right now, remembering that what got you to where you are now won’t necessarily get you to where you need to go next.
How to build a metrics culture that works
One of the biggest risks with metrics is that they get mandated from the top down and then people look for ways to game them.
For example, if the senior management wants to get a sense of the quality of the products, they might focus on measuring defect rates and whether that trend is going up or down. The development teams don’t want to get dinged, so they might choose not to check in anything until they know they will pass all the validations and meet the acceptance criteria. The metrics now look good, but there’s a negative impact on engineering practices and productivity.
This is why you have to be careful about how you use these things. Metrics really should be something that teams can use, meaning they’re meaningful for the product team that’s creating the metrics. It shouldn’t be a metric that senior management dictates, because ultimately that becomes a tax on the team. And I don’t know anybody that likes taxes, especially ones that don’t provide any value to them.
In the ideal state, you’re coaching teams to say, “Here are the things that your team should be looking at,” and the team should be developing metrics that are going to help them make decisions.
You want metrics to be created by the product teams and used by the product teams. And ultimately, you want senior managers to be able to use the information that development teams are producing, instead of dictating what metrics need to be provided.
The single focus principle
The four areas we’ve highlighted (inputs, estimates, quality and pipeline) aren’t independent – they’re deeply interconnected. Poor inputs mess with estimation. Bad estimates affect ability to plan. Both poor inputs and estimates impact the quality of the work and efficiency of the team. And those challenges ultimately show up in pipeline performance.
Here’s my takeaway for product people: figure out the one thing that’s going to help you incrementally improve. In general, it’s difficult for people to try and improve too many things at the same time or change too many things at the same time.
Going back to the golf analogy: if you’re trying to improve your golf swing and you change three things in your swing, you don’t know which one actually helped you improve. So typically you want to make one change at a time and see if it helps or not.
Measure what matters
Most product organizations are measuring themselves into paralysis (or they’re not measuring anything at all and going by feel). They have dashboards full of numbers that don’t drive decisions or improvement. The solution isn’t more metrics, it’s better metrics.
Start with the four fundamental questions:
- How good are your inputs?
- How good are your estimates?
- How productive are your resources?
- How good is your pipeline?
Pick one area where you most need improvement, measure it consistently, and focus on making incremental progress.
Remember, the goal isn’t to measure everything perfectly. It’s to measure the right things, at the right level of precision for your organization’s maturity, in a way that actually helps your teams get better at building products that customers value.
At the end of the day, the only metric that matters is the one that measures whatever is holding you back.
If you’re ready to move beyond vanity metrics and start measuring what actually drives product success, the first step is understanding where your organization stands today. At Ten Mile Square, our typical consulting engagements start with an assessment – looking at how an organization is doing and what their current state of things really is. We use these principles to identify those measurements and get to the root of what’s working and what isn’t. Contact us to learn more about our technology assessment process.