Test Strategy
Strategy is what we will test and why. A plan is how, who, and when. Know the difference — your stakeholders and your CTAL-TM exam both expect you to.
1 The Hook — Why This Matters
In 2021, a Wellington government agency rolled out a new RealMe integration for their citizen portal. The project plan was immaculate: sprints mapped, testers assigned, environments booked. But on launch day, users with screen readers could not complete authentication. The team had tested functionality exhaustively, yet nobody had written a strategy that mandated accessibility compliance against the NZ Web Accessibility Standard.
The fix was simple — add ARIA labels and keyboard navigation. The cost was a ministerial enquiry, a two-week delay, and a public apology. The plan was perfect for what it covered. The strategy was silent on what mattered most.
As a Test Lead, your strategy is the document that prevents this. It sets the guardrails before the sprint planning begins.
2 The Rule — The One-Sentence Version
A test strategy defines what will be tested, what will not be tested, and why — at the product level, for the long term.
It is not a schedule. It is not a resource spreadsheet. It is the reasoning layer that every test plan, sprint, and release inherits. When someone asks "why are we automating regression but not UAT?" the answer lives in the strategy.
3 The Analogy — Think Of It Like...
A city's building code versus a single construction schedule.
The building code says every commercial building needs sprinklers, wheelchair access, and earthquake bracing. It doesn't say when the concrete truck arrives. A single construction schedule says when the concrete truck arrives, but it doesn't decide whether sprinklers are needed. The strategy is the building code: long-lived, product-level, and challenged before the first brick is laid. The plan is the schedule: project-level, per-release, and derived from the code.
4 Watch Me Do It — Step by Step
Here is how to build a test strategy for a real NZ government project. Follow these steps for any product you lead.
- Define scope — what is in and what is out Be explicit. "In scope: functional, non-functional, security, and accessibility testing. Out of scope: penetration testing (owned by InfoSec), disaster recovery (owned by Ops)." Every out-of-scope item needs an owner.
- Choose test levels and types Decide which levels apply (unit, integration, system, UAT) and which types (functional, non-functional, structural, regression). For the RealMe project: all four levels, with mandatory accessibility and security types.
- Select your approach Manual, automated, exploratory, model-based? State the mix and the rationale. Example: regression suite automated for speed; UAT manual with real users; security via external penetration test.
- Define environments and tools Environment parity must be enforced. If staging doesn't mirror production, state the gap and the risk. List tools: test management, automation framework, CI/CD pipeline, defect tracker.
- Set entry and exit criteria Entry: build deployed, smoke tests pass, test data ready, environment stable. Exit: all critical defects resolved, coverage targets met, sign-off from product owner and security lead.
- Assign roles and deliverables Who owns the strategy? Who reviews it? What deliverables does it produce: test plan, defect reports, coverage metrics, release certification?
| Component | Decision | Rationale |
|---|---|---|
| Scope | Functional, accessibility, security, performance | Government mandate + public trust |
| Test levels | Unit, integration, system, UAT | High-risk citizen-facing system |
| Approach | Automated regression + manual UAT + external pen test | Speed + human judgement + specialist depth |
| Environment | Staging mirrors prod; weekly refresh | Environment parity reduces escape defects |
| Exit criteria | 0 critical open, 95% automated pass, accessibility audit green | Non-negotiable compliance gates |
5 When to Use It / When NOT to Use It
✅ Use a test strategy when...
- Starting a new product or major programme
- Stakeholders need confidence in quality approach
- Multiple teams share the same codebase
- Compliance or regulatory requirements exist
- You need a stable reference for per-release plans
❌ Don't use a test strategy when...
- You need a sprint schedule (that's a plan)
- The project is a single one-off hotfix
- The team has no authority to enforce it
- It is treated as a one-time sign-off document
- It copies another project's strategy without adaptation
Before you write a test strategy, ask:
- Does the project scope justify a strategy document, or is it one small feature?
- Will stakeholders and teams actually read and enforce it, or is it compliance theatre?
- Do you have buy-in from development, product, and leadership, or are you writing from testing alone?
- Can the strategy adapt to scope changes, or is it so rigid it becomes obsolete?
6 Common Mistakes — Don't Do This
🚫 Writing a strategy that is actually a plan
I used to think: a strategy should include dates, names, and sprint allocations.
Actually: That's a test plan. A strategy that includes Gantt charts loses its longevity. When the sprint changes, the strategy looks obsolete. Keep strategy at the "what and why" level. Let plans handle "how, who, and when."
🚫 Copying the last project's strategy
I used to think: if it worked for Project Alpha, it'll work for Project Beta.
Actually: Every product has different risk profiles, stakeholders, and compliance needs. A fintech strategy needs fraud testing. A healthtech strategy needs patient-data safeguards. Copy-paste strategies miss the product-specific risks that matter most.
🚫 No stakeholder review or quarterly refresh
I used to think: once the strategy is signed off, my job is done.
Actually: An unreviewed strategy becomes fiction. Products evolve, tech stacks change, and regulations update. Involve dev, ops, and security in the review. Schedule a quarterly refresh. A living strategy is credible; a forgotten one is shelfware.
When this technique fails
Test strategies fail when they are written in isolation by testing and then handed down to developers without input. They also fail when they are signed off once and never reviewed: as the product evolves, the strategy becomes fiction. Finally, they fail when they are so ambitious or complex that the team ignores them in favour of pragmatic shortcuts.
7 Now You Try — Strategy Audit
Scenario: A startup building an NZ healthcare patient portal writes a test strategy. It covers functional testing, unit tests, and regression automation. It does not mention data privacy, penetration testing, or accessibility.
Question: What is the single biggest gap, and what should the strategy mandate instead?
Biggest gap: no compliance or non-functional coverage
Healthcare in NZ is bound by the Privacy Act and Health Information Privacy Code. The strategy must mandate privacy impact assessment, penetration testing, and accessibility compliance (Web Accessibility Standard). Without these, the strategy is not fit for the product's risk profile. A good lead challenges the strategy before the first line of code is written.
8 Self-Check — Can You Actually Do This?
Click each question to reveal the answer. If you got all three, you're ready for practice.
Q1. What is the difference between a test strategy and a test plan?
Strategy = what and why. It is product-level, long-lived, and defines scope, levels, types, approach, and exit criteria. Plan = how, who, and when. It is project-level and per-release, containing schedules, resources, and task assignments. The plan inherits from the strategy.
Q2. Why should a test strategy explicitly state what is OUT of scope?
Because ambiguity becomes blame. If the strategy does not say "penetration testing is out of scope, owned by InfoSec," a production security flaw becomes testing's fault by default. Explicit out-of-scope items with named owners protect the team and set stakeholder expectations.
Q3. How often should a test strategy be reviewed, and who should be involved?
Review quarterly at minimum, and after any major product pivot, tech stack change, or regulatory update. Involve test leads, developers, DevOps, security, and the product owner. A strategy reviewed only by testers is a strategy that will be ignored by everyone else.
9 Interview Prep — Lead-Level Q&A
Q. What's the difference between a test strategy and a test plan?
A test strategy defines what we will test and why at the product level. It is long-lived, stable across releases, and covers scope, levels, types, approach, environment, tools, entry/exit criteria, roles, and deliverables. A test plan defines how, who, and when for a specific project or release. It contains schedules, resource assignments, task breakdowns, and sprint-level detail. The plan inherits from and must align with the strategy.
Q. How do you handle a stakeholder who wants to remove accessibility testing from the strategy to save time?
I would present the risk in business terms: non-compliance with the NZ Web Accessibility Standard exposes the organisation to reputational damage, legal challenge, and exclusion of users. I would propose alternatives such as prioritising critical user journeys for accessibility first, or using automated accessibility scanning to reduce manual effort. If the stakeholder still insists, I would document the decision, the associated risk, and the sign-off in the strategy's risk register. The strategy must reflect reality, but reality must be consciously chosen, not accidentally ignored.
Q. What components do you include in a test strategy for a high-risk government integration?
Scope with explicit in/out items; all four test levels (unit, integration, system, UAT); test types including functional, non-functional, security, and accessibility; a blended approach of automated regression, manual UAT with real users, and external penetration testing; environment parity requirements; tool selection with rationale; entry and exit criteria tied to compliance gates; role definitions and RACI; and a deliverables list including coverage reports and release certification. I would also include a review schedule to keep the strategy current.
Q. How do you keep a test strategy from becoming shelfware?
Treat it as a living document. Schedule quarterly reviews with cross-functional attendees. Update it when the product, tech stack, or regulations change. Reference it during sprint planning and release readiness reviews. If the team never opens the strategy after day one, it is not a strategy — it is a checkbox. Make it useful by connecting it to real decisions: "Does this release need UAT? Check the strategy. Do we automate this? Check the strategy."