Process Improvement
Using data to drive change. Moving the needle on organisational maturity and engineering performance.
1 The Hook
A Test Manager in an Auckland fintech was told: "Our testing is too slow." Instead of guessing, they measured their **DORA metrics**. They discovered that while testing took 2 days, the *wait time* for an environment took 5 days. By fixing the environment bottleneck, they halved the lead time without changing a single test case. Process improvement is about finding the "Wait" in the "Work."
Senior engineer insight
The hardest thing I learned about process improvement is that the most visible waste is rarely the most expensive waste. Every team I've worked with thought their testing was the bottleneck — until we mapped the value stream and found the real culprit sitting in a queue: waiting for a build, waiting for an environment, waiting for a sign-off. What shifted my thinking entirely was running a three-hour value stream mapping workshop with a Wellington government agency and watching the room go silent when they realised 78% of their lead time was pure wait, not work. TMMi gives you a ladder to climb, but the value stream map shows you which floor you're actually on.
The most common mistake Test Manager-level testers make: they target a TMMi level as the goal, rather than treating it as a diagnostic tool — and end up gold-plating processes that nobody in the organisation has the capacity to sustain.
From the field
A central government agency in Wellington was undergoing a CMMI Level 3 assessment and had invested heavily in documented test processes — templates, sign-off gates, the lot. The assumption was that documentation maturity equalled process maturity. What the assessment revealed was that teams were filling in the templates retrospectively, after release, because nobody had time during sprints. The processes existed on paper but had zero adoption at the team level. The real fix wasn't more documentation — it was running fortnightly retrospectives where teams fed pain points directly into an improvement backlog that the Test Manager owned and reported on quarterly. Within two quarters, environment-related delays dropped by 40% because the teams themselves were surfacing the problems. The lesson: process improvement is a cultural artefact, not a document set.
2 Maturity Frameworks (TMMi)
The **Test Maturity Model integration (TMMi)** provides a roadmap for organisational growth.
| Level | Name | Characteristics |
|---|---|---|
| Level 1 | Initial | Testing is chaotic, ad-hoc, and purely reactive. |
| Level 2 | Managed | Basic test planning and environment control established. |
| Level 3 | Defined | Testing is a standardised part of the Dev lifecycle (Shift-Left). |
| Level 4 | Measured | Quality metrics are tracked and used to predict outcomes. |
| Level 5 | Optimising | Continuous, data-driven improvement and defect prevention. |
3 Engineering Performance (DORA)
Modern QE functions align with DORA metrics to prove their impact on the delivery pipeline.
Deployment Frequency
How often we ship. High-performing teams ship multiple times per day with automated safety nets.
Lead Time for Changes
How long from "Code Commit" to "Production." Testing should be a highway, not a toll booth.
Change Failure Rate
What % of releases cause an issue? This is the ultimate measure of testing effectiveness.
Time to Restore
How fast we fix prod issues. Dependent on your BCP and test environment readiness.
4 Improvement Artifacts
Documents that track the "Journey to Better."
Maturity Assessment
A "Gap Analysis" showing where you are today vs. TMMi standards. Used to secure budget for improvements.
DORA Dashboard
A real-time view of engineering health. Shifts the conversation from "Bugs" to "Pipeline Health."
Improvement Backlog
A prioritised list of process fixes (e.g. "Automate environment spin-up", "Implement TDD").
Value Stream Map
A visual diagram of the test process identifying "Value" time vs. "Waste" time.
Why teams fail here
- Chasing TMMi level numbers as status symbols rather than solving the specific bottlenecks that are actually costing delivery time — the level is a byproduct of real improvement, not the target.
- Measuring DORA metrics without baselining first — teams celebrate a Drop in Change Failure Rate from 22% to 18% while the industry median they're being compared against is 5%, making the improvement look good on a slide while the underlying problem remains severe.
- Running retrospectives that produce action items nobody owns — without a named Test Manager tracking improvement backlog items to completion in the next sprint, retros become venting sessions with no feedback loop.
- Presenting process improvement as a testing initiative when the root causes live in environments, deployments, or change management — this loses stakeholder buy-in the moment you ask another team to change their behaviour without evidence framed in their language (delivery speed, not defect counts).
Key takeaway
Process improvement at the Test Manager level is not about upgrading your documentation — it is about finding where time disappears and building the organisational habit of naming it, measuring it, and eliminating it one retrospective at a time.
5 Self-Check
Q1. Is TMMi Level 5 the goal for every company?
No. Level 5 is expensive. Most NZ organisations find Level 3 (Defined) provides the best balance of quality and cost.
Q2. How does testing improve "Lead Time for Changes"?
By automating regression and environment setup, removing the need for 2-week "Testing Phases" before release.