Test Metrics & Reporting
Metrics turn testing activity into evidence. A test lead who cannot quantify quality cannot make the case to ship or hold. This page covers the KPIs that matter, the difference between leading and lagging indicators, how to present to stakeholders, and how to avoid the traps that make metrics misleading.
1 The Hook
A test lead at a NZ insurer stands up at the release meeting with a slide they are proud of: “We ran 2,400 test cases this release, up 30% on last time.” The room nods. The release ships. Two weeks later, claims customers can’t upload supporting documents — a core journey nobody had a test for. The 2,400 number was real, but it answered the wrong question.
The big test-count slide measured effort, not quality, and it could not influence a single decision. It did not say whether the risky areas were covered, whether the P1 defects were closed, or whether requirements had tests at all. Worse, chasing a high test count had quietly encouraged shallow, duplicated tests — which is exactly how a whole journey ends up with zero coverage while the headline number climbs.
This is the pattern behind misleading test reports: the team measures what is easy to count instead of what would change a decision. A number that cannot move a go/no-go call is decoration, not evidence — and decoration on an executive slide is actively dangerous, because it buys false confidence.
The most dangerous test report I ever signed off on showed a 96% pass rate and zero open P1s. We shipped. Three days later a NZ government portal was returning incorrect benefit entitlement calculations for edge-case income brackets — a journey nobody had a test for. The 96% was real; it just measured the wrong 96%. What I hadn't checked was escaped defect history: two of the three previous releases had also produced post-go-live incidents in that same calculation module. The data existed. It was just in a separate spreadsheet nobody opened during the release meeting. Since then my rule is: before any release sign-off, pull the escaped defect trend for the last three cycles. A single metric in isolation is a story with no context. Three releases of escaped defects tells you whether your test process actually works.
Senior engineer insight
The shift that changed how I think about test metrics: stop building dashboards for yourself and start building them for the person who has to make a decision at 9am on release day. Once I started asking "what would make a release committee member either worried or confident right now?", I stopped reporting test-case counts entirely and started leading with open-blocker trends and requirements coverage on the riskiest module. The numbers got smaller and the conversations got sharper.
Most common mistake: reporting metrics in the past tense. By the time you present escaped-defect counts from last release, the current release has already shipped. The metrics that protect the programme are the ones you report while there is still time to act.
From the field
On a large NZ government programme — multi-agency, nine-month cycle, executive steering group reviewing quality monthly — the test lead inherited a dashboard that had never changed: test cases run, test cases passed, defects open. Three releases in, the steering group had developed a comfortable habit of nodding at the green cells and moving on. Then a release went live with a defects-per-transaction rate three times higher than the previous version. The dashboard had shown green because nobody had defined what “open defects” meant: seventeen P2s had been quietly reclassified as “deferred enhancements” to keep the cell green. The fix wasn’t technical — it was governance: independent defect triage with a BA from outside the test team, agreed threshold definitions documented in the test plan, and a new dashboard column showing “deferred enhancements” as its own line so it couldn’t absorb real risk. The lesson that generalises: a metric is only as honest as the process that produces it, and in NZ programme reporting, the pressure to show green is real enough that you need structural protection, not just good intentions.
2 The Rule
A metric only earns its place if it is tied to a decision and shown with a target and a trend — report leading indicators that let you act in time, not vanity counts that merely describe effort.
3 The Analogy
The dashboard in your car, not the odometer alone.
Driving from Auckland to Wellington, the odometer telling you “you have driven 400 km” is a vanity number — it measures effort, not whether you will arrive. The gauges that matter are the fuel level, the temperature, and the time-to-destination, because each one can change a decision while you still have road left: stop for petrol, pull over before the engine cooks, or pick a faster route. A warning light is worth more than any distance count.
Test metrics work the same way. “We ran 2,400 tests” is the odometer. Requirements coverage, open P1 count, and execution rate against plan are the fuel gauge and warning lights — leading indicators that tell you where you are heading while there is still time to act. A good RAG dashboard is a car dashboard, not a trip counter.
Common Mistake vs What Works
Reporting test case pass rate as the headline quality metric. The slide goes to the release committee: “95% pass rate — looking good.” Nobody asks what the 5% failures are, which requirements are in that 5%, or why the suite has grown from 800 tests to 1,200 without a coverage audit. The number is real; it just answers the wrong question — and it actively discourages the follow-up questions that would reveal the risk.
Report metrics tied to decisions, not test activity. Lead with defect escape rate (how many bugs reached production this cycle and the trend), mean time to detect critical defects, and requirements coverage of new user stories. Show pass rate as a health signal for the test suite itself — one input alongside the numbers that actually tell you whether the product is ready to ship. Pair every figure with a target, a trend, and a plain-language interpretation so stakeholders can act, not just nod.
What it is
Test metrics are quantitative measurements that describe the state of testing at a point in time or over a period. They serve two purposes: internal (how is the team tracking against plan?) and external (what should stakeholders believe about quality?). Both purposes require different metrics and different presentations.
Good metrics are objective, reproducible, and tied to a decision. If a metric cannot influence a decision — go/no-go, increase test effort, delay release, reduce scope — it is a vanity metric. Collect the data, but do not present it as if it matters.
ISTQB definition: “A test metric is a measurement derived from test activities and the test basis, used to support decisions and improvements in the test process.” Metrics without decisions are just data.
Core KPIs
Defect Detection Rate (DDR)
The number of defects found per unit time (per sprint, per week, per test cycle). A rising DDR early in a cycle is expected and healthy — the team is finding problems. A DDR that fails to decline as the project matures is a red flag: either new defects are being introduced faster than they are being fixed, or the scope of testing has changed.
Defect Removal Efficiency (DRE)
DRE = (Defects found before release) / (Defects found before release + Escaped defects) × 100%
DRE is the percentage of total defects that were caught before reaching production. Industry benchmarks vary by domain: financial systems typically target 95%+; enterprise software 85–95%. A DRE below 80% indicates the test process is not catching defects effectively. Measure DRE per release and track the trend.
Escaped Defects
Defects that reached production and were reported by users or monitoring systems. Each escaped defect is a data point about what the test process missed. Classify escaped defects by: severity, which test phase should have caught it, and why it was missed (not covered? wrong priority? wrong technique?). Use escaped defect analysis to improve test design, not just to report the number.
Test Execution Rate
Tests executed per day as a percentage of tests planned. A rate below 100% of plan is a warning that the team will not finish execution by the deadline. Track this daily during execution phases and escalate early when the rate drops — waiting until the end of the cycle to discover 30% of tests were not run is too late to recover.
Test Pass Rate
Passing tests as a percentage of executed tests. A pass rate climbing steadily towards an agreed threshold (e.g., 95% P1/P2 pass rate) is a healthy exit criteria signal. A pass rate that plateaus well below the threshold signals that defects are not being fixed fast enough relative to execution pace.
Defect Density
Defect density = Defects found / Size of component (function points, KLOC, story points)
Normalises defect counts by the size of the component, enabling comparison across modules of different sizes. High defect density in a specific module is a red flag for that module’s quality — investigate whether to increase test depth, request a code review, or flag it in the risk register.
Requirements Coverage %
The percentage of requirements (user stories, acceptance criteria, use cases) with at least one test case. A coverage gap means untested requirements — functionality the team has agreed to deliver but has not verified. Requirements coverage is a leading indicator of release risk: low coverage early in the cycle means the team needs to accelerate test design, not just execution.
Leading vs lagging indicators
The distinction between leading and lagging indicators is critical for useful reporting:
- Lagging indicators measure outcomes after the fact. Escaped defects, DRE, and post-release customer-reported issues are lagging. They tell you how you did — useful for retrospectives but not for preventing the current release’s problems.
- Leading indicators signal future outcomes while there is still time to act. Requirements coverage %, test execution rate against plan, and open blocker defect count are leading. They tell you where you are heading so you can intervene.
Stakeholder dashboards should prominently feature leading indicators. Lagging indicators belong in retrospectives and quality improvement plans.
Presenting to stakeholders
Different stakeholders need different cuts of the same data:
- Engineering team — defect counts by component, open vs closed trends, execution rate vs plan. Granular, daily.
- Product owner — requirements coverage %, open P1/P2 defects by feature area, estimated days to exit criteria. Weekly.
- Senior leadership / release committee — overall RAG status, key risks, recommendation (ship / hold / ship with known issues). High-level, per release.
Traffic light (RAG: Red/Amber/Green) dashboards work well for executive stakeholders because they force the test lead to make a judgement call, not just present data. If the pass rate is 93% and the threshold is 95%, is that Amber or Green? The test lead needs to own that decision and justify it.
Trend charts are more informative than point-in-time snapshots. A defect count of 40 open issues is meaningless without knowing whether that number is rising or falling. Always show the trend alongside the current value.
Worked example: sprint test report
| KPI | Target | Actual | Trend | Status |
|---|---|---|---|---|
| Test execution rate | 100% by end of sprint | 94% | ↑ (was 81% mid-sprint) | AMBER |
| Test pass rate (all) | ≥ 90% | 92% | ↑ improving | GREEN |
| P1/P2 pass rate | 100% | 97% | — stable | AMBER |
| Open P1 defects | 0 at release | 1 open | ↓ (was 3) | RED |
| Requirements coverage | ≥ 95% | 98% | ↑ improving | GREEN |
| Defect density (new module) | ≤ 2 defects/story point | 4.1 | ↑ rising — flag | RED |
This report tells a clear story: the sprint is close to exit criteria but cannot ship with the open P1 defect. The new module’s defect density (4.1 vs a target of 2) is a signal that either the module needs more testing depth or a targeted code review. The test lead’s recommendation is: hold release until the P1 is resolved and investigate the new module defect density before the next sprint.
When metrics lie — Goodhart’s Law in testing
Goodhart’s Law: “when a measure becomes a target, it ceases to be a good measure.” Applied to testing:
- Pass rate gaming — if the team is incentivised to hit a 95% pass rate, failing tests may be marked as “known issues” or “environment problems” to inflate the number. Validate pass rate data independently.
- Test count inflation — measuring team productivity by the number of test cases written leads to shallow, redundant tests. Measure coverage and defect detection, not test count.
- Zero-defect pressure — teams pressured to report zero open defects find creative ways to redefine what a defect is. Track defect trends with consistent definitions and independent triage.
- Escaped defect undercounting — escaped defects are only known if production monitoring is in place. A team with no monitoring can claim perfect DRE simply because no one is measuring what escapes.
The best protection against metrics gaming is to define every metric’s calculation, data source, and threshold before the cycle begins, and to have those definitions reviewed by someone outside the test team.
ISTQB mapping
| Syllabus ref | Topic | Level |
|---|---|---|
| CTAL-TM Ch. 4 | Defect management metrics — DDR, DRE, defect density, escaped defects | Advanced / Lead |
| CTAL-TM Ch. 5 | Test metrics — coverage, execution, pass/fail rates, stakeholder reporting | Advanced / Lead |
| CTFL 5.3 | Test monitoring and control — basic metrics awareness | Foundation |
Metrics mastery is primarily a Test Lead — CTAL-TM — topic. Foundation candidates need awareness that testing generates metrics and that they are used to monitor and control the test process. Advanced candidates must be able to define, collect, interpret, and present test metrics for a real project.
Common mistakes
- No baseline to compare against — a pass rate of 87% is meaningless without knowing what was expected. Establish targets and historical baselines before the cycle begins.
- Measuring what is easy, not what matters — test case count and execution rate are easy to measure. Escaped defects and DRE are harder but more meaningful. Invest in the harder measurements.
- Vanity metrics in executive reports — “we ran 2,000 tests” is not a quality signal. “We achieved 98% requirements coverage with a P1/P2 pass rate of 100%” is.
- Reporting metrics without context — always accompany a metric with a trend, a target, and an interpretation. A number on its own forces the reader to guess what it means.
- Updating metrics only at the end of the cycle — metrics are most useful as leading indicators during execution. Daily or at least weekly updates allow intervention while there is still time to act.
4 Industry Reality
- Metrics definitions are inherited, not designed. Most teams inherit a spreadsheet from a previous lead with columns labelled "Pass %", "Defects Open", and "Tests Run" — no targets, no trend history, no explanation of what "Pass" even means. Your first job is usually not to report metrics but to define them from scratch mid-project.
- Stakeholders anchor on the wrong number. A release committee that has seen "Tests Run: 2,400" for six months will ask about that number even after you've replaced it with requirements coverage. Changing reporting culture takes sustained effort — you may need to run both the old vanity slide and the new decision-driving dashboard in parallel for several releases before the old one gets dropped.
- DRE is almost never tracked in practice. Calculating Defect Removal Efficiency requires good production monitoring and consistent defect classification — two things many NZ teams lack. Escaped defects often don't get logged against the release that missed them, so the real DRE number is unknown. Senior testers substitute with trend analysis on customer-reported production incidents instead.
- Pass rate gaming is real and widespread. Under release pressure, test leads face subtle pressure to reclassify blocked or failing tests as "environment issues" or "deferred to next release." The textbook answer is independent triage; the real-world answer is building that habit before the crunch, not during it, because mid-sprint you won't win the argument.
- Tools fragment the data. Execution rate might live in Jira, defect counts in Azure DevOps, and coverage percentages in a manual spreadsheet. Producing a coherent dashboard means stitching together sources that don't agree on column names, date formats, or what counts as "closed." Automation helps, but most teams do this manually every sprint.
5 When to Use It — and When Not To
✓ Use it when
- You need to make or support a release go/no-go decision and require objective evidence of quality.
- The project has well-defined requirements or acceptance criteria that can serve as a coverage denominator.
- You are running a test cycle long enough (more than a few days) that trends are meaningful — a one-day smoke test doesn't need a dashboard.
- Stakeholders are sceptical of the "trust us, quality is fine" approach and need data to make risk decisions.
- You are tracking test-team productivity or process improvement across multiple releases and need comparable data points over time.
✗ Skip it when
- The team is tiny (1–2 testers) and the project short — the overhead of setting up and maintaining metric dashboards exceeds the value.
- You lack a stable requirements baseline — coverage % is meaningless if the denominator keeps changing mid-sprint.
- You're doing pure exploratory testing where the goal is discovery, not execution against a fixed test set.
- No one will act on the numbers — if stakeholders are going to ship regardless of what the metrics say, you're producing reports nobody reads.
- The measurement infrastructure isn't in place — collecting accurate metrics manually across fragmented tools creates more noise than signal.
Context guide
How the right level of test metrics effort changes based on project context.
| Context | Priority | Why |
|---|---|---|
| Benefits NZ or Revenue NZ benefit/tax system — high-volume government release with an executive steering group | Essential | Errors affect citizens' financial entitlements. The steering group must see formal RAG dashboards, requirements coverage, and escaped defect trends before every release gate. Informal status updates are not sufficient governance. |
| Harbour Bank or Pacific Bank core banking platform — regulated financial system under FMA oversight | Essential | Regulatory obligations require documented evidence of test coverage and defect management. DRE and escaped defect data feed directly into audit evidence and must be archived per release. |
| HealthNZ patient-facing portal or clinical decision system | Essential | Patient safety risk means every metric must be defined before the cycle begins and reviewed by someone outside the test team. Defect density on clinical calculation modules is non-negotiable to track — not just the aggregate pass rate. |
| Pacific Air or Spark — mid-size agile team running two-week sprints on a customer-facing product | High | Real-time dashboards showing execution rate vs plan, open blocker count, and requirements coverage per sprint are enough. DRE is replaced by escaped defect trend from production monitoring — most NZ agile teams track incidents, not formal DRE. |
| TransitNZ minor CMS content update or internal admin tool with a three-day test window | Medium | A lightweight checklist of pass/fail against acceptance criteria is sufficient. A full RAG dashboard would cost more to produce than the value it returns — match the measurement investment to the risk profile. |
| Start-up or internal hackathon prototype — no production users, no regulatory exposure | Low | Formal metrics overhead exceeds value. A shared Jira board showing open bugs by severity is more than enough. Invest the saved time in exploratory testing instead. |
Trade-offs
What you gain and what you give up when you choose test metrics and reporting.
| Advantage | Disadvantage | Use instead when… |
|---|---|---|
| Turns subjective "testing is going well" into objective evidence a release committee can act on — go/no-go becomes a data-driven call rather than gut feel. | Metrics are only as honest as the process that produces them. Under deadline pressure, teams reclassify failures as "environment issues" or redefine what counts as a defect — the dashboard shows green while risk accumulates. | Stakeholder trust in the team is already high and the project is short — use a verbal status update and a shared defect board rather than a formal dashboard that will be misread or gamed. |
| Leading indicators (requirements coverage %, execution rate vs plan, open blocker count) give the test lead time to intervene before problems compound — the earlier you see a red trend, the more options you have. | Defining, collecting, and stitching together metrics across Jira, Azure DevOps, and manual spreadsheets consumes tester time that could be spent testing — especially in teams without automated reporting pipelines. | You are running pure exploratory testing with no fixed test set — execution rate vs plan is meaningless when there is no plan denominator. Use session-based test management notes instead. |
| Historical metric archives (DRE trend, escaped defect rate per release) build an evidence base for process-improvement conversations — you can show the steering group that quality is genuinely improving over multiple releases, not just this one. | Stakeholders anchor on the first metrics they see. Changing a dashboard — even to better metrics — requires sustained effort across multiple releases before the old vanity numbers stop being the first question in the room. | The requirements baseline changes every sprint (common on agile builds at Spark or Harbour Bank) — coverage % becomes a moving target. Track defect escape rate per sprint instead, which does not depend on a stable denominator. |
| Tailored reports for different audiences (engineering, product owner, release committee) train each group to read only what is relevant to them — decisions get faster and stakeholder noise reduces. | Goodhart's Law is always lurking: once a metric becomes a target, behaviour warps around it. Test count, pass rate, and zero open defects are the three most commonly gamed metrics in NZ programmes. | No one will act on the numbers — if the release committee ships regardless of what the metrics say, switch energy to risk-based testing coverage of the most dangerous journeys rather than producing a dashboard nobody reads. |
Enterprise reality
How test metrics changes at 200–300-developer scale in NZ enterprise
- Metric collection is automated and governed, not manual. At Pacific Bank or Revenue NZ scale, dashboards pull directly from Jira, Azure DevOps, and Xray via API — a tester producing a weekly spreadsheet is a compliance gap, not a workaround. Automated pipelines own defect counts, execution rates, and pass/fail trends; the test lead's job shifts to interpreting the output and escalating anomalies, not assembling the numbers.
- Compliance obligations dictate what you must measure. Under the Privacy Act 2020, NZISM, and PCI DSS, organisations like Coastal Bank and Southern Bank cannot treat DRE and escaped defect data as optional — they are audit artefacts. Every release gate produces a signed-off metrics package that is retained and available for review by the FMA or RBNZ. A dashboard that "looks fine" is not enough; it must be independently signed off by a QA lead who is not part of the delivery team.
- Tooling operates at a volume that changes the game. Organisations like CloudBooks and ListRight run test suites of 50,000+ automated cases; defect density reports are generated per commit, not per sprint, using tools such as Allure, TestRail, or Zephyr Scale integrated into CI/CD. At this scale, a human-curated RAG slide is replaced by live quality gates that block a pipeline build if pass rate drops below threshold — the metric becomes an enforcement mechanism, not a reporting one.
- Cross-squad metric alignment across 10+ teams is where organisations fall over. When 12 squads at a programme like HealthNZ's digital infrastructure uplift each define "defect closed" differently, the aggregate DRE figure is meaningless. Enterprise QA leads spend significant effort on metric standardisation — enforcing shared defect classification taxonomies, common severity definitions, and consistent sprint boundaries — because without it the portfolio-level dashboard is fiction. Getting it wrong means a programme sponsor approves a go-live decision based on an aggregate that hides three squads running at a 60% pass rate.
◆ What I would do
Professional judgement — when to reach for test metrics and reporting, when to skip it, and what to watch for.
The bottom line: Build your dashboard for the person who has to make a decision at 9am on release day. If a metric cannot change what that person does next, it does not belong on the slide.
6 Best Practices
- ✓ Define every metric before the cycle starts. Agree on the calculation, data source, and threshold with the product owner and engineering lead before a single test is run — not mid-sprint when numbers are inconvenient.
- ✓ Always pair a number with a target and a trend. Never report a bare count. "42 open defects" is meaningless; "42 open defects, target 0 by Thursday, down from 71 last week" is a story.
- ✓ Lead with leading indicators. Put requirements coverage %, execution rate vs plan, and open blocker count at the top of stakeholder dashboards — they're the fuel gauge, not the trip counter.
- ✓ Tailor the cut to the audience. Engineering gets granular daily data by component. Product gets coverage by feature area and estimated days to exit criteria. Executives get a RAG status and a clear recommendation. Never send the same report to all three.
- ✓ Track escaped defects by root cause, not just count. When a defect escapes to production, classify it: was it not covered? wrong priority? wrong technique? Trend those root causes to drive process improvement, not just blame attribution.
- ✓ Use defect density to spot hot modules. A module with 4x the defect density of comparable modules is a risk signal worth escalating to the engineering team for a targeted code review — don't let the aggregate test-team pass rate hide it.
- ✓ Update metrics at least weekly, preferably daily during execution. Metrics are most valuable as leading indicators during the execution phase. A dashboard updated only at the end of the sprint is retrospective, not actionable.
- ✓ Build in independent triage. Defect classification and pass/fail decisions should have a second set of eyes — a BA, product owner, or senior tester outside the team being measured. This is the practical protection against Goodhart's Law.
- ✓ Show the RAG threshold explicitly. Don't make stakeholders guess why something is Amber. "P1/P2 pass rate 97% — Amber because threshold is 100%" is unambiguous and keeps you honest.
- ✓ Archive metric data per release. Historical baselines let you show whether quality is improving release over release. DRE trending from 82% to 91% over six months is a compelling story; having that data is what makes the story possible.
7 Common Misconceptions
❌ Myth: A high pass rate means high quality.
Reality: A high pass rate only means the tests that were run mostly passed. It says nothing about what wasn't tested, whether the tests were meaningful, or whether the pass/fail threshold was applied honestly. A 99% pass rate with 30% requirements coverage and no production monitoring is not evidence of quality — it's evidence that the team ran a lot of tests and most of them passed. Combine pass rate with requirements coverage, defect density on high-risk modules, and escaped defect history before drawing any conclusions about quality.
❌ Myth: More test cases means more thorough testing.
Reality: Test count is a vanity metric. Teams chasing a high test-case count produce shallow, duplicated tests that inflate the number without improving coverage or defect detection. Two well-designed boundary-value tests on a payment calculation will find more defects than twenty superficial "click through the happy path" tests. Measure requirements coverage %, defect detection rate, and defect density — not how many rows are in your test management tool.
❌ Myth: Zero open defects at release means the product is ready to ship.
Reality: Zero open defects is one of the most gamed metrics in testing. Under release pressure, defects get reclassified as "known issues", "deferred to backlog", "environment-only", or "by design" — all legitimate labels in some cases, all frequently misused. Zero open defects is only meaningful alongside: consistent defect classification with independent triage, requirements coverage showing the tested scope, and escaped defect history from previous releases showing the team's actual track record. Without those companion metrics, zero open defects is a number, not a quality signal.
8 Now You Try
Three graded exercises — spot, fix, then build. Write your answer, run it for AI feedback, then compare to the model answer.
For each metric below, label it vanity, leading, or lagging, and say what decision (if any) it can drive: (a) total test cases written this sprint; (b) requirements coverage %; (c) escaped defects last release; (d) open P1 defect count; (e) defect removal efficiency (DRE).
Show model answer
(a) Total test cases written — VANITY. Measures effort, not quality, and drives no decision. Worse, chasing it encourages shallow, duplicated tests. (b) Requirements coverage % — LEADING. Low coverage early signals release risk while there is still time to add test design. Decision: accelerate test design, or accept the gap. (c) Escaped defects last release — LAGGING. Measured after the fact; useful for retrospectives and improving test design, but cannot prevent the current release's problems. (d) Open P1 defect count — LEADING (and a live blocker). Drives the go/no-go: you cannot ship with an open P1. Show the trend (rising or falling) alongside the count. (e) DRE — LAGGING. Calculated after escapes are known; drives process-improvement decisions across releases, not the current go/no-go. Key idea: stakeholder dashboards should feature leading indicators (coverage, execution rate, open blockers); lagging indicators belong in retrospectives; vanity metrics should not be presented as if they matter.
The release-committee slide below is misleading. Rewrite it into a small set of decision-driving metrics, each with a target, the actual, and a trend, and end with a clear ship / hold recommendation. Invent plausible numbers for a NZ KiwiSaver provider.
“Quality is great. We ran 3,100 tests (a record!) and the pass rate is 99%. Zero defects open. Recommend ship.”
Rewrite as a decision-driving report:
Show model answer
A decision-driving rewrite (numbers invented but plausible): Metric — Target — Actual — Trend — Status - Requirements coverage — ≥ 95% — 88% — flat — AMBER (12% of requirements have no test; risk of untested functionality) - P1/P2 pass rate — 100% — 96% — improving — AMBER - Open P1 defects — 0 at release — 1 open — down from 3 — RED (cannot ship with an open P1) - Escaped defects last release — trend down — 4, up from 1 — rising — RED (test process is missing defects; investigate) Recommendation: HOLD. The open P1 alone blocks release; the coverage gap and rising escape rate compound the risk. Resolve the P1, close the coverage gap on the highest-risk requirements, then re-assess. What I distrust about the original: "99% pass rate" with "zero defects open" is a Goodhart's Law red flag — when a measure becomes a target, teams find ways to hit it (marking failures as "known issues" or "environment problems", redefining what counts as a defect). The 3,100 test count is a vanity metric. And "zero open defects" is meaningless without escaped-defect data and independent triage — you cannot claim quality without measuring what reached production.
You are the test lead on a NZ government online services release. Design the metrics you would report to three different audiences — the engineering team, the product owner, and the release committee — explaining why each cut suits that audience. Then name one safeguard you would put in place against metrics gaming (Goodhart’s Law).
Show model answer
A strong reporting set: Engineering team — defect counts by component, open vs closed trends, execution rate vs plan; granular, daily. Why: they act on specifics and need to see which component is unstable and whether execution is on track to finish. Product owner — requirements coverage %, open P1/P2 defects by feature area, estimated days to exit criteria; weekly. Why: they own scope and priority and need to know which features are at risk and when "done" is realistic. Release committee — overall RAG status, the key risks, and a clear recommendation (ship / hold / ship with known issues); high-level, per release. Why: they make the go/no-go and need a judgement call backed by leading indicators, not raw data. Always pair each number with a target and a trend. Safeguard against gaming: define every metric's calculation, data source, and threshold BEFORE the cycle begins, and have those definitions reviewed by someone outside the test team. Validate pass-rate data independently and keep consistent defect definitions with independent triage, so failures cannot be quietly reclassified as "known issues" to hit a target. Measure coverage and defect detection, not test count.
Why teams fail here
- Measuring what is easy to count (test cases run, total defects logged) rather than what influences decisions (requirements coverage, open blocker trend, escaped defect rate) — the dashboard fills up with numbers that nobody acts on.
- Presenting metrics without a target or trend — "42 open defects" is meaningless without knowing the target was zero and the trend is rising; a bare number forces the reader to guess what it means, and they will usually guess wrong.
- Defining metrics mid-cycle under release pressure — when the threshold for "pass" is set after results are in, Goodhart's Law activates immediately; teams reverse-engineer the threshold to match the data rather than use the data to assess readiness.
- Sending the same report to all audiences — engineering needs daily defect counts by component; the release committee needs a RAG status and a single recommendation; a combined report serves neither and trains both groups to stop reading it.
Key takeaway
A metric earns its place only if it is tied to a decision, shown with a target and a trend, and reported while there is still time to act on what it says.
How this has changed
The field moved. Here is how Test Metrics evolved from its origins to current practice.
Test metrics are defect counts and test pass/fail percentages. Reports go to project managers who use them for status tracking. Metrics are trailing indicators — they show what happened, not what is at risk. Goodhart's Law applies immediately: teams game pass rates by relaxing pass criteria.
ISTQB and IEEE formalise test metrics taxonomy. Defect density, coverage percentage, test execution progress, and defect removal efficiency become standard metrics. Dashboards proliferate. The problem: more data does not produce better decisions.
Agile forces a rethink. Teams with two-week sprints cannot wait for monthly metrics reports. Real-time dashboards, CI build health, and team-level metrics replace project-level reporting. "Leading vs lagging indicators" becomes a key distinction.
DORA metrics (Deployment Frequency, Lead Time, MTTR, Change Failure Rate) provide engineering metrics that correlate with business outcomes. Quality metrics begin integrating with engineering performance metrics rather than existing separately.
AI tools can surface anomalies in test metric trends — flagging when a metric moves in an unexpected direction before it becomes a problem. The maturity model for metrics has three levels: measure what is easy, measure what matters, and use metrics to drive improvement conversations (not report status).
Self-Check
Click each question to reveal the answer.
Interview Questions
What NZ hiring managers ask about Test Metrics — and what strong answers look like.
What metrics do you track to show the quality of a test suite, not just the coverage percentage?
Strong answer: I track: mutation score (do tests detect injected defects?), flakiness rate (percentage of tests that fail non-deterministically), test execution time trend (is the suite getting slower?), test-to-code ratio (tests growing proportionally to code?), and defect escape rate (percentage of defects found after release vs before). Coverage is a one-dimensional view — these metrics together give a multi-dimensional picture of suite health. A suite with 80% coverage and 5% mutation score is weaker than a suite with 70% coverage and 90% mutation score.
Mid/Senior
Your team is under pressure to show value to stakeholders. Which testing metrics would you present, and which would you avoid?
Strong answer: I would present: defect escape rate trend (direct evidence of preventing production incidents), test execution time trend (evidence of CI efficiency), and mean time to detect (how quickly we find regression after a code change). I would avoid presenting raw defect count (higher is not better — it reflects what was found, not prevented), coverage percentage alone (misleading without mutation context), and test case count (more tests is not better — maintaining fewer, effective tests is the goal). I frame metrics around business outcomes: "we prevented 14 defects from reaching production last quarter" is more compelling to stakeholders than "we have 3,200 test cases."
Senior/Lead
Q1: What single test decides whether a metric is worth reporting?
Can it influence a decision — go/no-go, increase test effort, delay release, reduce scope? If a metric cannot change a decision, it is a vanity metric. Collect the data if it is cheap, but do not present it as if it matters.
Q2: What is the difference between a leading and a lagging indicator, and where does each belong?
Lagging indicators measure outcomes after the fact (escaped defects, DRE, post-release issues) — useful for retrospectives but too late to fix the current release. Leading indicators signal future outcomes while there is still time to act (requirements coverage %, execution rate vs plan, open blocker count). Stakeholder dashboards should feature leading indicators; lagging ones belong in retrospectives.
Q3: Why is a raw number like “40 open defects” nearly useless on its own?
Without a target and a trend it forces the reader to guess what it means. 40 open defects falling from 80 is good news; 40 rising from 10 is alarming. Always show a metric with its target, its trend, and an interpretation — a number alone has no meaning.
Q4: State Goodhart’s Law and give one way it distorts testing metrics.
“When a measure becomes a target, it ceases to be a good measure.” Example: if the team is incentivised to hit a 95% pass rate, failing tests get reclassified as “known issues” or “environment problems” to inflate the number. Other forms: test-count inflation producing shallow tests, and zero-defect pressure leading teams to redefine what counts as a defect.
Q5: What is the best protection against metrics gaming?
Define each metric’s calculation, data source, and threshold before the cycle begins, and have those definitions reviewed by someone outside the test team. Validate pass-rate data independently, keep consistent defect definitions with independent triage, and measure coverage and defect detection rather than test count.
Q: Your team is testing the Benefits NZ online benefit application portal before a major release. You have three weeks of execution left and your test lead asks which two metrics should drive the daily standup conversation. Which do you choose and why?
A: Open blocker defect count and test execution rate against plan are the two to prioritise. Open blockers (P1/P2 defects) directly threaten release readiness on a government service where inaccessibility has real consequences for benefit recipients. Execution rate against plan tells you whether the team will finish — if you are at 60% of plan halfway through week one, you need to either reduce scope or add resource before the problem compounds. Requirements coverage % is a close third, but at execution stage the blockers and pace are the most actionable leading indicators in a daily stand-up context.
Q: What is the key difference between Defect Detection Rate (DDR) and Defect Removal Efficiency (DRE), and when does each metric become useful?
A: DDR measures the rate at which defects are being found over time (defects per sprint or per week), making it a useful in-cycle leading indicator — a DDR that fails to decline as the project matures signals new defects are being introduced faster than they are being fixed. DRE measures the proportion of all defects that were caught before release (defects found pre-release divided by total defects including escapees), making it a lagging quality indicator calculated only after production data is available. DDR guides decisions mid-cycle; DRE informs cross-release process improvement and is most meaningful when tracked as a trend over several releases. In practice, DRE is rarely tracked in NZ teams because it requires good production monitoring and consistent defect classification across the full post-release window.
Q: A developer on an Revenue NZ tax-filing system says “We have 98% test pass rate and zero open defects — we’re clearly ready to ship.” What is wrong with this reasoning and how do you respond?
A: The statement confuses a high pass rate with high quality — a classic Goodhart’s Law trap. A 98% pass rate only tells you that 98% of the tests that were run passed; it says nothing about what was not tested, whether the tests were meaningful, or whether defects were reclassified as “known issues” to hit the number. “Zero open defects” is equally suspect without independent triage and a consistent defect definition — under release pressure on an Revenue NZ system, failing tests can be quietly moved to “deferred to backlog” or “environment-only.” The right response is to ask: What is the requirements coverage percentage? What does the escaped defect history from previous releases look like? Who independently triaged the defects that were closed or reclassified? On a critical government system where filing errors can attract penalties, these companion metrics are not optional — they are what give the pass rate meaning.
Q: When should you NOT invest in setting up a formal test metrics dashboard, even if you have the data available?
A: Skip the formal dashboard when the overhead exceeds the value: a team of one or two testers on a short project, pure exploratory testing where there is no fixed test set to measure execution rate against, or a situation where the requirements baseline keeps changing mid-sprint (making coverage % a moving target). Most importantly, skip it when no one will act on the numbers — if the release committee is going to ship regardless of what the metrics say, producing a RAG dashboard consumes time that would be better spent testing. In NZ agency contexts, a one-week smoke test before a minor CMS update does not need a DRE calculation; a three-month platform migration for TransitNZ or HealthNZ does. Match the investment in measurement to the risk profile and the length of the cycle.
Related techniques
Metrics are most useful when combined with a risk lens. Use Risk-Based Testing to decide which areas to measure most carefully — high-risk areas need more granular metrics than low-risk areas.
Exit criteria in Test Planning should be expressed as metrics thresholds. The test plan defines “what does done look like?” — metrics provide the evidence that the answer is yes.
Defect metrics (DDR, DRE, escaped defects) are produced by and feed back into the Defect Management process. Root cause analysis on escaped defects is particularly valuable for improving both the test process and the development process.
Practice this technique: Try Test Lead Practice 07 — Test coverage gaps, Test Lead Practice 08 — Defect triage.