Test Metrics & Reporting
Evidence over intuition. Metrics turn testing activity into data that stakeholders can understand. A Test Lead who can't quantify quality can't defend their decisions.
1 The Hook — The "Feeling" Trap
A stakeholder asks you: "How is testing going?" You say: "I think it's going well, we've found quite a few bugs." This is a trap. "Good" and "Quite a few" are feelings, not facts. A developer might think 10 bugs is a lot, while a product owner thinks it's a little.
Professional Leads use Metrics to remove the emotion. "We have 95% requirements coverage and a 98% pass rate" is a fact that nobody can argue with. Metrics turn your hard work into evidence.
2 The Rule — DDR, DRE, and Density
A Lead's dashboard should track the Efficiency of the team and the Density of the bugs.
- DDR (Defect Detection Rate): How many bugs per day are we finding?
- DRE (Defect Removal Efficiency): What % of total bugs did we find *before* production? (Target >95%).
- Defect Density: Bugs per Feature/Component. Where is the "trash fire"?
3 The Analogy — The Car Dashboard
Speedo vs. Fuel Gauge.
Imagine driving from Auckland to Wellington. Your **Speedometer** (Test Execution Rate) tells you how fast you're moving *right now*. Your **Fuel Gauge** (Remaining Bugs) tells you how much work is left before you run out of energy. If your Speedo is high but your Fuel is low, you're going to break down before you arrive. A Test Lead needs both: Leading indicators (Speed) to see if they'll finish on time, and Lagging indicators (Fuel) to see how much quality is left in the tank.
4 Watch Me Do It — NZ App Defect Density
Scenario: You're testing a new NZ Real Estate app with three modules.
| Module | Size (Stories) | Bugs Found | Defect Density |
|---|---|---|---|
| Search | 10 | 5 | 0.5 |
| Payments | 5 | 20 | 4.0 |
| Profile | 5 | 2 | 0.4 |
The Insight: The "Payments" module has a density 10x higher than the others. Even if the dev says "it's finished," the metrics tell you this is a high-risk area that needs more testing. Numbers don't lie.
5 Decision Tool — Leading vs. Lagging
⏱ Leading (The Future)
- Execution Rate: Are we on schedule?
- Requirements Coverage: Did we miss any features?
- Goal: Prevent failure before the deadline.
📑 Lagging (The Past)
- Escaped Defects: What did we miss?
- DRE %: How efficient were we?
- Goal: Learn for the next project.
6 Common Mistakes
🚫 Vanity Metrics (Test Count)
I used to think: "We ran 2,000 tests!" sounds impressive.
Actually: Who cares? If 1,999 of those tests were trivial checks, you haven't proven anything. Stakeholders care about **Coverage** and **Risk**, not the total number of clicks.
🚫 Reporting without Context
I used to think: I'll just send the defect chart.
Actually: 40 bugs found is "Good" if it was 100 yesterday. It's "Bad" if it was 0 yesterday. Always show the **Trend** (Is it going up or down?).
7 Now You Try — Calculate the DRE
Scenario: You found 90 bugs during testing. After release, users found 10 more bugs.
What is your Defect Removal Efficiency (DRE)?
Math: 90 / (90 + 10) = 90 / 100 = 90%.
8 Self-Check
Q1. What is Goodhart's Law?
"When a measure becomes a target, it ceases to be a good measure." If you reward testers for finding many bugs, they will log trivial "typo" bugs just to hit their targets.
Q2. What is a "RAG" status?
Red, Amber, Green. It is the most important part of a report. You use the metrics to justify the color. "We are RED because our P1 pass rate is only 40%."
9 Interview Prep
"How do you handle a team gaming the metrics?"
Answer: "I use balancing metrics. If I'm measuring the number of bugs found (to encourage deep testing), I also measure the 'Rejection Rate' (to ensure bugs are valid). If I'm measuring speed, I also measure 'Escaped Defects' (to ensure quality isn't sacrificed for speed). This prevents the team from focusing on only one number."
10 Next Step
You've measured the quality. Now, let's learn how to manage the problems you've found. Next: Defect Management.