Test Planning
A test plan is the contract between the test team and the project: what will be tested, how, by whom, and with what success criteria. Without it, testing is improvised; with a bad one, testing is theatre. This page covers what a plan must contain, entry and exit criteria, estimation, and the critical distinction between a test strategy and a test plan.
1 The Hook
A health-sector team in Auckland is two days from releasing a patient-booking update. The testers have been heads-down for a fortnight and feel good about it. Then someone asks the question nobody planned for: "How do we know we're done?" Silence. Nobody had written down what "done" meant.
So testing ends the way it usually does without a plan — the calendar runs out. The release ships. A week later a defect surfaces in the appointment-reminder flow, an area everyone assumed "someone" had covered. They had not. It was never in scope, because scope was never written down. There was no entry criterion to confirm the environment was ready, no exit criterion to confirm coverage, and no risk register flagging the reminder integration as fragile.
None of this was a testing-skill failure. The testers were good. What was missing was the contract: an agreed statement of what would be tested, when testing could start, and what "finished" looked like. Without it, testing becomes improvisation, and improvisation under deadline is where gaps hide.
A test plan is the agreed contract for a testing effort — it makes scope, entry criteria, and exit criteria explicit and signed off before testing starts, so "done" is an objective decision rather than the moment the calendar runs out. Use it whenever scope is ambiguous, testers are coordinating across streams, or you need a documented artefact for regulatory or stakeholder sign-off. The most common mistake is defining exit criteria but not entry criteria — without entry criteria, testing starts against broken builds, wastes effort, and produces coverage metrics that cannot be trusted.
The risk register is where test plans die in practice. Every plan I have reviewed in NZ government and health projects has one. Almost none of them actually drive test depth. Teams write three or four risks, assign them a likelihood/impact score, and then the test cases look identical to what they would have built anyway. The register becomes a sign-off artefact, not a planning tool. The check I apply: take your top two risks and count how many test cases directly target them. If the answer is "a couple extra," the risk register is theatre. High-risk areas should have three to five times the test coverage of low-risk ones. If the numbers do not reflect the risk scores, rewrite the test design before you circulate the plan.
2 The Rule
A test plan is the agreed contract for a testing effort — it must make scope (in and out), entry criteria, and exit criteria explicit and signed off before testing starts, so that "done" is an objective decision, not the moment the calendar runs out.
3 The Analogy
A building consent and code-compliance certificate.
Before a NZ builder pours a foundation, the council issues a consent that says exactly what is being built and to what standard. Before anyone moves in, a Code Compliance Certificate confirms the work met that standard. The consent is the entry criterion — you do not start without it. The certificate is the exit criterion — the job is not "done" because the builder feels finished; it is done when the agreed conditions are demonstrably met and signed off.
A test plan plays both roles at once. Its entry criteria are the consent: do not begin testing until the environment, build, and data are ready. Its exit criteria are the compliance certificate: testing is complete when the defined conditions are met, not when the deadline arrives. Skip either and you are the builder who poured concrete with no consent and handed over the keys with no certificate.
Common Mistake vs What Works
The test plan is written at project kick-off, presented to stakeholders, and then filed. By delivery, the actual testing bears no resemblance to it — scope has shifted, risks have changed, and the entry/exit criteria were never enforced. The plan becomes a compliance artefact rather than a working guide, and everyone quietly ignores it while improvising testing under deadline pressure.
Write a lightweight test strategy at the start — one to two pages covering scope, approach, risks, and entry/exit criteria. Then update it at each sprint retrospective as reality changes. A test plan that lies is worse than no plan at all: it creates false confidence in coverage that does not exist. Keep it short, keep it honest, and keep it current. In NZ government and health projects, this living document is also your audit trail if a defect surfaces post-release.
What it is
A test plan is a document (or set of lightweight artefacts in agile) that defines the scope, approach, resources, schedule, and criteria for a testing effort. Its primary purpose is to make the test team’s intent visible and agreed — by stakeholders, developers, product owners, and management — before testing begins.
Planning is not bureaucracy. Even in a two-person team working in two-week sprints, someone needs to have thought through: what are we testing, when are we done, what happens if we run out of time, and who is responsible for what. A test plan documents those decisions so they are explicit and agreed, not implicit and assumed.
ISTQB definition (CTFL 5.1): “Test planning defines the test objectives and the approach to achieve them within the constraints of the project.” Test planning is a continuous activity — the plan is updated as the project evolves, not written once and filed.
Test strategy vs test plan
These are distinct artefacts that are frequently confused:
- Test strategy (organisational level) — a high-level document that defines the organisation’s standard approach to testing: which testing levels are performed (unit, integration, system, acceptance), which techniques are preferred, how regression is managed, what automation tooling is standard. It applies across multiple projects. It changes infrequently.
- Test plan (project level) — a specific document for a specific project or release. It derives from the strategy but specifies scope, schedule, resources, risks, and criteria for this project. It changes as the project changes.
In practice, many agile teams do not maintain a separate strategy document — it lives in the team’s working agreements. The test plan is then the only formal planning artefact, and it should at minimum include the items listed below.
What a test plan must contain
IEEE 829 defines the classic structure (now IEEE 29119-3). Whether you follow the full standard or a lightweight agile variant, these elements must be addressed somewhere:
- Scope — what is in scope and what is explicitly out of scope. Both matter. Out-of-scope items prevent scope creep and make explicit what the team is not responsible for testing.
- Test approach / strategy — which testing techniques will be applied, at what levels (unit, integration, system, acceptance), manual or automated, and in what sequence.
- Entry criteria — conditions that must be met before testing can begin. See the section below.
- Exit criteria — conditions that define when testing is complete. See the section below.
- Resources — who is doing the testing, what test environments are needed, what data, what tools.
- Schedule — timeline for test design, environment setup, execution phases, and reporting milestones.
- Risks and contingencies — what could go wrong (environment not ready, requirements incomplete, resource unavailable) and what the response plan is. Derived from a risk register.
- Deliverables — what will be produced: test cases, test reports, defect reports, sign-off document.
Entry and exit criteria
These are the most important and most frequently omitted elements of a test plan.
Entry criteria define the pre-conditions for testing to start. Common criteria:
- Feature code is complete and reviewed (no obvious compile errors, PR merged)
- Test environment is available and verified as stable
- Test data is in place (accounts, products, reference data seeded)
- Unit tests pass (typically ≥ agreed coverage threshold)
- Requirements/acceptance criteria are documented and signed off
If testing starts before entry criteria are met, the test team wastes time testing against incomplete or broken builds. Define entry criteria and enforce them.
Exit criteria define when testing is complete. Common criteria:
- All P1 defects resolved and verified closed
- P2 defects resolved or risk-accepted with documented justification
- Test execution rate: 100% of planned test cases executed
- Test pass rate: ≥ 95% overall, 100% of critical path tests
- Requirements coverage: ≥ 98% of acceptance criteria have at least one passing test
- Regression suite: passing at agreed threshold
- Sign-off received from product owner or release authority
Without exit criteria, testing can end for one of two bad reasons: you ran out of time, or someone said “I think it’s good enough.” Exit criteria make the decision objective and auditable.
Suspension criteria — a third criterion type worth adding. Suspension criteria define when to pause testing due to a blocker (e.g., a P1 defect that makes 30% of test cases impossible to execute). Define these upfront so the team knows when to stop rather than executing tests that produce invalid results.
Estimation in test planning
The test plan must include a time and effort estimate. Three common approaches:
- Work Breakdown Structure (WBS) — decompose testing into tasks (test design, review, environment setup, execution, defect retesting, reporting) and estimate each. Bottom-up, most accurate but most effort to produce.
- Metrics-based — use historical data from previous similar projects. “Last quarter’s release of similar scope took 80 person-hours of testing.” Requires good historical records.
- Expert judgement — consult experienced team members. Quickest but most subjective. Mitigate subjectivity by using planning poker or three-point estimation (optimistic, most likely, pessimistic) to average multiple expert views.
Always include assumptions in your estimate. “This estimate assumes the test environment is available from day 1 of the sprint” is more useful than the number alone, because it signals immediately when an assumption is violated and the estimate needs revision.
Worked example: login feature mini test plan
| Section | Content |
|---|---|
| Scope (in) | Username/password login, “Remember me” toggle, account lockout after 3 failures, password reset flow |
| Scope (out) | SSO / OAuth login (separate sprint), biometric login (separate workstream) |
| Approach | EP + BVA for input fields; decision table for lockout logic; exploratory for session edge cases; Playwright automation for regression |
| Entry criteria | Login feature code merged to staging; test environment stable; test accounts seeded (valid, locked, expired); acceptance criteria signed off |
| Exit criteria | 0 open P1/P2 defects; ≥ 95% pass rate; all AC covered; regression suite green; product owner sign-off |
| Risks | Risk 1: Auth library upgrade may change session behaviour — mitigate by running session tests last. Risk 2: Test environment uses a shared DB — data collisions possible; mitigate by using namespaced test accounts. |
| Estimate | Test design: 4h · Review: 1h · Execution: 6h · Defect retest: 2h · Reporting: 1h — Total: 14h (assumes stable environment from day 1) |
Agile test planning
In agile teams, the master test plan is often replaced by a combination of:
- Team working agreements — cover approach, tooling, and definition of done (which includes testing obligations)
- Sprint planning — covers scope and schedule for each sprint
- Per-story acceptance criteria — cover scope and exit criteria for each feature
- Test charters — cover approach for exploratory sessions
The content of a test plan still needs to exist — it is just distributed across these artefacts rather than in a single document. A common mistake on agile teams is to confuse “we don’t write formal test plans” with “we don’t plan testing.” The planning happens; it just takes a different form.
For larger agile releases (PI planning in SAFe, quarterly planning, major version releases), a lightweight release-level test plan is valuable even in highly agile environments. It documents the overall scope, the regression strategy, the automation coverage goals, and the release criteria — content that does not fit neatly into individual sprint planning.
ISTQB mapping
| Syllabus ref | Topic | Level |
|---|---|---|
| CTFL 5.1 | Test planning — purpose, content, and the planning process | Foundation |
| CTFL 5.1 K2 | Explain the purpose of a test plan and the typical content | Foundation LO |
| CTAL-TM Ch. 2 | Test management — full test planning framework, risk-based approach, estimation, strategy | Advanced / Lead |
| CTAL-TM 2.2 K4 | Create a test plan for a given project using a risk-based approach | Advanced LO |
Foundation candidates must be able to explain the purpose of a test plan and recall its typical contents. Advanced (CTAL-TM) candidates must be able to create a risk-based test plan for a given project scenario — this is a K4 (create) learning objective, the highest level in Bloom’s taxonomy.
Common mistakes
- Skipping exit criteria — without explicit exit criteria, testing ends when time runs out. This is the single most common cause of releasing with known gaps that “we didn’t have time to test.” Define exit criteria first, then plan the effort to achieve them.
- No risk register — every project has risks (incomplete requirements, environment instability, resource constraints). Document them in the plan with a mitigation action for each. Teams that don’t plan for risks are surprised when they materialise — and then improvise under pressure, which is the worst time to make testing decisions.
- Plan written once and never updated — a test plan written at sprint planning and not revisited is wrong by day 3. Update scope, risks, and estimates when things change. The plan is a communication tool, not a historical record.
- Conflating test strategy with test plan — the strategy says “we use risk-based testing for all projects.” The plan says “for this release, the high-risk areas are X, Y, and Z, and they will receive these specific test techniques.” Both are needed; neither substitutes for the other.
- Ignoring entry criteria — teams frequently define exit criteria but not entry criteria. Without entry criteria, testing starts before it should, wastes time on broken builds, and produces invalid results that distort the metrics.
4 Industry Reality
- Requirements arrive late and incomplete. Real test plans are written against half-finished specs. Senior testers document what is known, flag what is missing as a risk with an owner and a deadline, and version the plan as the requirements stabilise — rather than waiting for a perfect spec that never arrives.
- Nobody signs off entry criteria. The plan says "feature code merged and environment stable" but the sprint starts Monday regardless. In practice, testers negotiate: get verbal sign-off from the dev lead, log the environment instability as a defect immediately, and track it as a plan risk. You raise the problem — you do not silently absorb it.
- The plan is a formality in many NZ SME teams. At smaller NZ companies (health, fintech, government contractors) a "test plan" is often a Confluence page with a scope list and a Jira filter for exit criteria. That is fine. The content matters, not the format. What you rarely see outside enterprise is the full IEEE 29119-3 document — know the standard for ISTQB but apply lightweight versions at work.
- Estimation is usually wrong and everyone knows it. Testers give estimates, managers halve them, and scope expands. The senior tester's move: estimate with explicit assumptions documented, then use those assumptions to surface the conversation when scope changes. "We estimated 14 hours assuming a stable environment from day 1. The environment was unavailable for two days — the estimate is now 18 hours."
- Exit criteria are renegotiated under pressure. It happens on every project. A P2 defect is open at release date and someone asks to risk-accept it. Exit criteria make that conversation explicit and documented — instead of a silent decision by whoever has the most authority in the room. Testers who write good exit criteria are the ones who control that conversation, not the ones it happens to.
Senior engineer insight
The project that changed how I think about test planning was a TransitNZ road-levy system upgrade. We had a solid plan — scope defined, entry criteria agreed, exit criteria measurable. What I did not plan for was the in-scope boundary shifting three times in five days as business stakeholders kept adding “small” edge cases. The plan survived, but only because I had written scope-out items explicitly. Every new request was met with “that is out of scope per the agreed plan — here is the signed version.” Without that documented out-of-scope list, each addition would have quietly expanded the test window until we missed the regulatory deadline. The plan was not a bureaucratic exercise; it was the only thing that held the boundary.
The most common graduate mistake: writing entry criteria but never enforcing them — testing starts anyway because the sprint began, and two days later you are logging defects that are actually environment issues rather than real bugs.
From the field
On a Wellington City Council rates system upgrade, the test team inherited a two-year-old test plan from the previous vendor and assumed scope was settled. Three weeks into execution, a data migration stream appeared in the backlog — the vendor had considered it "out of scope by default," but the council assumed it was covered. Nobody had written it down either way. We spent four days reverse-engineering what had been tested, what hadn't, and who owned the gap. The fix was simple but late: a scope-out list agreed in writing at planning prevents the "I thought they were covering that" conversation from happening on the week before go-live. Document what you are not testing as carefully as what you are.
5 When to Use It — and When Not To
✓ Use it when
- You are coordinating testing across more than one tester, QA stream, or sprint — a shared written plan prevents duplicated and missed coverage
- The release has regulatory or contractual obligations (NZ Privacy Act 2020 compliance, health-sector audit requirements, government agency UAT sign-off) — you need a documented artefact
- Scope is large or ambiguous — a plan forces the conversation about what is explicitly in and out before testing starts
- You need to negotiate timelines or resources with management — a plan with estimates and assumptions is your evidence base
- You are onboarding a new team member or handing work to another tester mid-sprint — the plan is the briefing document
✗ Skip it when
- You are the sole tester on a two-day hotfix with clear, agreed acceptance criteria — the AC is your plan; do not gold-plate it
- The team already has mature working agreements, strong acceptance criteria per story, and a Definition of Done that covers testing — the planning content exists; adding a document adds no value
- You are in early prototyping or spike work where the feature may be thrown away — plan when the work is real, not speculative
- The "test plan" would be written after testing ends to satisfy a process checkbox — if it is retrospective, it is documentation, not planning; call it what it is
Context guide
How the right level of Test Planning effort changes based on project context.
| Context | Priority | Why |
|---|---|---|
| Regulated system (Revenue NZ, CoverNZ, Pharmac, Benefits NZ, health.govt.nz) | Essential | Formal plan is the audit artefact — documented scope, entry/exit criteria, and risk-accepted defects are required evidence under Privacy Act 2020 and agency governance frameworks; absence is a compliance exposure, not just a process gap. |
| Large NZ programme delivery (multi-vendor, multi-stream, AoG or council) | Essential | Scope boundaries between teams are the single most common source of untested integration paths; an explicit scope-in and scope-out list, signed off across vendors, is the only reliable defence against the “I thought they were covering that” gap. |
| Enterprise product release (coordinating multiple testers or QA streams) | High | A shared, living plan prevents duplicated coverage on some paths and zero coverage on others; it is also the negotiation tool when timelines are compressed and scope must be cut transparently. |
| Legacy system migration (lift-and-shift, re-platform, or data migration) | High | Scope is poorly understood at the start and grows quickly; entry criteria enforced on data-readiness prevent the most common migration failure (testing against incomplete or corrupted data sets); a risk register surfacing fragile integrations early prevents last-minute discovery crises. |
| Agile sprint delivery (mature team, stable working agreements, strong per-story AC) | Medium | Planning content already exists across the Definition of Done and backlog; a lightweight release-level plan (scope, regression strategy, release criteria) adds value at major version points but a full plan per sprint is overhead — keep it to the elements the team actually uses. |
| Small startup or new product build (single tester, fast-moving scope, clear AC per story) | Low | Scope changes daily; a formal plan becomes stale before you finish writing it. Invest effort in tight acceptance criteria and a clear Definition of Done instead — revisit when the team grows beyond two testers or the product reaches its first regulated milestone. |
Trade-offs
What you gain and what you give up when you choose Test Planning.
| Advantage | Disadvantage | Use instead when… |
|---|---|---|
| Makes “done” objective and auditable — exit criteria remove the subjective judgement call at release time | Takes time to write well; poorly written plans are worse than none — a vague plan gives false confidence | The team is small, scope is clear, and per-story acceptance criteria already define done for every piece of work |
| Explicit scope-out list prevents untested areas slipping through under the assumption “someone covered it” | Plans stale quickly — scope, risks, and estimates drift from day 3; maintaining accuracy requires discipline | A two-day hotfix with agreed, locked scope — the AC is your plan; adding a document adds no value |
| Gives testers negotiating power — a documented plan with assumptions lets you surface the conversation when scope expands or estimates need revising | Can create process theatre — teams write plans to satisfy an audit requirement and then ignore them entirely | The team has mature working agreements and a living Definition of Done that already answers every planning question |
| Creates an audit trail for regulated environments (Privacy Act, CoverNZ, Revenue NZ, Pharmac) — documented risk-acceptance protects the tester and the organisation | Entry criteria enforcement requires organisational will — if stakeholders override them anyway, the plan is ceremonial | Early prototype or spike work where the feature may be thrown away — plan when the work is real, not speculative |
| Risk register drives test depth allocation — high-risk areas get proportionally more coverage when the plan is used properly | Risk registers are routinely written and then ignored — test cases end up uniform regardless of the risk scores | A single-tester exploratory session on a well-understood feature — a test charter gives the same coverage decision faster |
6 Best Practices
- ✓ Write scope out-of-scope first. Explicitly listing what is NOT being tested prevents the most common planning failure — untested areas everyone assumed "someone" had covered.
- ✓ State exit criteria before estimating effort. Decide what "done" looks like, then plan backwards to the effort needed to get there. Reversing this order produces estimates that satisfy the schedule, not coverage.
- ✓ Document assumptions alongside every estimate. "This estimate assumes environment availability from day 1 and stable test data" converts a number into a conversation starter when conditions change.
- ✓ Add suspension criteria to every plan. Define the blocker threshold (e.g. a P1 that blocks >25% of test cases) at which testing pauses. Without it, teams execute tests that produce meaningless results.
- ✓ Treat the plan as a living document. Schedule a brief plan review at each sprint planning or weekly — scope, risks, and estimates drift constantly. A plan last updated on day 1 is wrong by day 3.
- ✓ Link the risk register to test depth. Identify the three to five highest-risk areas and assign proportionally heavier test coverage. Document this explicitly so stakeholders can see where effort is going and why.
- ✓ Get entry criteria accepted in writing before the sprint starts. A Slack message confirming "environment will be available Monday" counts. The point is: when it is not available Monday, you have evidence — not just a verbal recollection.
- ✓ Use three-point estimation for uncertain areas. Optimistic / most likely / pessimistic averaged with weighting (O + 4M + P) / 6 produces a range, not a false-precision number. Ranges are more honest and more useful to project managers.
- ✓ Reference the test strategy rather than restating it. If your organisation has a standard approach to regression, automation tooling, or defect severity definitions — link to it, do not copy it. Duplication creates version-control debt.
- ✓ Finish the plan with a sign-off block. Named approvers and their sign-off date turn the plan into an agreed contract. Without names, the plan is a suggestion.
7 Common Misconceptions
❌ Myth: Agile teams do not need test plans — sprint planning replaces them.
Reality: Sprint planning covers scope and schedule for one sprint; it does not address cross-sprint regression strategy, environment dependencies, release criteria, or resource risks. The content of a test plan must exist somewhere in every project — on agile teams it is distributed across working agreements, Definition of Done, acceptance criteria, and release planning rather than in a single document. Saying "we are agile, we do not plan testing" confuses the format with the activity.
❌ Myth: A test plan written at project start is complete and does not need updating.
Reality: A test plan written once and filed is wrong almost immediately. Requirements change, environments are delayed, resources move, risks materialise. Senior testers update scope, risks, and estimates continuously — at minimum once per sprint — and communicate changes to stakeholders. A plan that does not reflect current reality is worse than no plan, because it gives false confidence about coverage and timelines.
❌ Myth: Exit criteria are too rigid for real projects — you always have to ship regardless.
Reality: Exit criteria are not a veto; they are a structured decision framework. When the release deadline arrives and P2 defects are open, exit criteria make the risk-acceptance conversation explicit and auditable — "we are releasing with these three known defects, accepted by [product owner] on [date]." Without exit criteria, the same decision happens invisibly. Documented risk acceptance is not failure; undocumented shipping of unknown quality is.
8 Now You Try
Three graded exercises — spot, fix, then build. Write your answer, run it for AI feedback, then compare to the model answer.
A test plan for a KiwiFirst Bank online statement download lists the conditions below. For each, label it as an entry criterion, an exit criterion, or a suspension criterion, and say why.
Show model answer
a) Test environment available and verified stable — ENTRY. A pre-condition that must be met before testing can begin; without it you test against a broken or unavailable build. b) All P1 defects resolved and verified closed — EXIT. A condition that defines when testing is complete. c) A defect blocks 30% of test cases from running — SUSPENSION. A blocker that means you should pause testing rather than execute tests that produce invalid results. d) Acceptance criteria documented and signed off — ENTRY. You cannot meaningfully test against requirements that are not agreed. e) Product owner sign-off received — EXIT. The final condition confirming the testing effort is formally complete. The pattern: entry = "can we start?", exit = "are we done?", suspension = "should we pause?".
A tester wrote the exit criteria below for a release. They are too vague to be auditable. Rewrite them as specific, measurable exit criteria.
• Most of the tests pass
• No really bad bugs left
• The team feels good about it
Rewrite as measurable exit criteria:
Show model answer
Specific, measurable exit criteria: - 100% of planned test cases executed. - Test pass rate ≥ 95% overall, and 100% of critical-path tests passing. - 0 open P1 defects; all P2 defects resolved or formally risk-accepted with documented justification. - ≥ 98% of acceptance criteria covered by at least one passing test, and regression suite passing at the agreed threshold; product owner sign-off received. Why the originals were unusable: - "Most of the tests pass" has no number — most could mean 60% or 95%. Exit criteria must state a threshold. - "No really bad bugs left" has no severity definition — define it by priority (0 open P1) so it is objective. - "The team feels good about it" is a feeling, not a criterion. Exit criteria exist precisely to remove subjective judgement so the done decision is auditable.
A NZ supermarket is releasing an online click-and-collect ordering feature. Build a lightweight test plan summary covering: scope (in and out), approach, entry criteria, exit criteria, and two risks with mitigations.
Show model answer
Lightweight test plan — click-and-collect ordering: Scope (in): browse and add groceries to a collection order, select store and collection time slot, substitutions handling, payment, order confirmation and amendment before cut-off. Scope (out): home delivery (separate workstream), loyalty-points redemption (separate sprint). Approach: EP + BVA on quantity and time-slot fields; decision table for substitution rules; exploratory on the slot-availability and cut-off logic; automated regression on the existing cart and checkout flow. Entry criteria: feature merged to staging; environment stable; test accounts and store/slot data seeded; acceptance criteria signed off. Exit criteria: 100% of planned tests executed; ≥ 95% pass rate, 100% of critical-path (order → pay → confirm) tests passing; 0 open P1/P2 defects; regression suite green; product owner sign-off. Risk 1: time-slot capacity is shared real-time data — concurrent orders could oversell a slot. Mitigation: include concurrency tests and run them against an isolated store with controlled slot capacity. Risk 2: substitution rules are complex and easily misread from the spec. Mitigation: build a decision table for the substitution logic first and review it with the product owner before executing. A senior would call out the cut-off-time boundary and the slot-oversell concurrency case as the highest-risk areas and weight test depth there.
Enterprise reality
Programmes with multiple workstreams, governance committees, and formal sign-off
- Master test plans for large programmes (e.g. an AoG ERP rollout or an Revenue NZ core-tax replacement) require approval from a project board or Programme Governance Committee before testing begins — sign-off is not optional and can take weeks; plan for it in the schedule.
- Phase test plans for each release stream are subsidiary documents, each with their own scope, entry/exit criteria, and sign-off authority — the master plan sets the boundaries; each phase plan details what happens within them.
- Entry and exit criteria become contractual at programme scale — a workstream that fails to meet exit criteria before the integration test phase can block every other stream; missing an entry criterion is a programme-level risk, not just a team issue.
- Test plans are auditable artefacts at this scale — reviewed by internal assurance, the Office of the Auditor-General, or external programme assurance reviewers; every scope-out decision and every risk-accepted defect needs documented justification from a named authority.
Why teams fail here
- Writing entry criteria but never enforcing them — the sprint starts Monday regardless, so testers begin execution against a broken environment and spend two days logging defects that are actually infrastructure noise.
- Treating the risk register as a sign-off artefact rather than a test-depth driver — three risks are documented, assigned a colour, and then the test cases look identical to what they would have been without a risk register at all.
- Defining only what is in scope and leaving out-of-scope implicit — on large NZ government programme deliveries across multiple agencies or vendors, the unwritten assumption is always "someone else covers that," which is exactly how critical integration paths go untested.
- Writing the plan at kick-off and never revisiting it — by week two of a complex council or health-system upgrade the scope, environment, and team have all shifted; a plan last updated on day one is fiction, not a contract.
Self-Check
Click each question to reveal the answer.
How this has changed
The field moved. Here is how Test Planning evolved from its origins to current practice.
IEEE 829 standard defines the Master Test Plan. Waterfall projects have formal, comprehensive test plans written before development begins. Plans are long, detailed, and often stale by the time testing starts.
Agile Manifesto published. "Working software over comprehensive documentation" challenges heavy test planning. Lightweight test strategies replace multi-hundred-page plans in many agile teams. IEEE 829 becomes less relevant to practitioners.
The industry splits: regulated domains (health, finance, government) keep formal planning for audit evidence; agile teams adopt lightweight team charters and sprint-by-sprint test approaches. Both use risk-based scoping.
ISO/IEC/IEEE 29119 replaces IEEE 829 with a process-agnostic standard. Controversy follows — some practitioners argue it imposes waterfall thinking on agile teams. The "Stop 29119" campaign highlights the tension between standards and agile practice.
AI tools can generate test plan drafts from requirements and risk registers. Human leads still determine strategy, scope decisions, and stakeholder agreements. NZ government programmes require test plans as gateway review evidence regardless of delivery methodology.
Q1: What is the single most common cause of releasing with known gaps, and which part of the plan prevents it?
Testing ending because time ran out rather than because coverage was achieved. Explicit, measurable exit criteria prevent it — they make "done" an objective, auditable decision instead of "I think it's good enough" or "the calendar said so".
Q2: What is the difference between a test strategy and a test plan?
A test strategy is organisational — the standard approach to testing across many projects (levels performed, preferred techniques, regression and automation standards); it changes rarely. A test plan is project-specific — it derives from the strategy but defines scope, schedule, resources, risks, and criteria for one project or release, and it changes as the project changes.
Q3: Why does scope need to state what is explicitly out of scope, not just what is in?
Out-of-scope items prevent scope creep and make explicit what the team is not responsible for testing. Stating only what is in scope leaves gaps that everyone assumes "someone" covered — which is exactly how an untested area slips through to production.
Q4: What is a suspension criterion and how does it differ from an exit criterion?
A suspension criterion defines when to pause testing because of a blocker — for example a P1 defect that makes a large share of test cases impossible to run. An exit criterion defines when testing is complete. Suspension says "stop for now"; exit says "we are finished". Defining suspension upfront stops the team executing tests that would only produce invalid results.
Q5: Is "we are agile, we don't write test plans" the same as "we don't plan testing"?
No. The content of a test plan still has to exist — scope, approach, entry/exit criteria — it is just distributed across working agreements, sprint planning, acceptance criteria, and test charters rather than living in a single document. Confusing "no formal document" with "no planning" is a common and costly mistake.
Q6: Your team is testing a new myIR dashboard feature for Revenue NZ. The environment has been available for two days but the acceptance criteria are still being finalised by the product owner. Should testing begin?
No — incomplete acceptance criteria is a violation of the entry criterion "requirements/acceptance criteria documented and signed off." Beginning testing now means testers will design cases against a moving target, waste effort retesting when criteria change, and produce coverage metrics that cannot be trusted. The right move is to raise this as a plan risk with a named owner and a deadline, and to use the time for test design and review rather than execution. In a government context like Revenue NZ, undocumented scope assumptions also create audit risk.
Q7: What is the key difference between test planning and risk-based testing, and how do the two techniques interact?
Test planning is the overall process of defining scope, approach, criteria, resources, schedule, and risks for a testing effort — it is the container. Risk-based testing is a specific technique for prioritising what to test by likelihood and impact of failure — it is one input into that container. The risk register in the test plan is populated using risk-based testing analysis; the results then drive how test depth is allocated across features. You cannot do risk-based testing without a plan to put the priorities into, and a plan without a risk register is missing its most important input.
Q8: A developer tells you that writing a test plan is unnecessary because the team has strong acceptance criteria on every story and a clear Definition of Done. What is right about this and what is missing?
The developer is right that per-story acceptance criteria and a solid Definition of Done cover a great deal of planning content — scope and exit criteria per feature are effectively distributed across the backlog. What those artefacts typically do not cover is cross-sprint regression strategy, environment dependencies and risks, resource allocation across multiple testers, release-level exit criteria, estimation, and sign-off authority. For a small team on routine sprints the developer's approach may be sufficient; for a major release, onboarding a new tester, a regulatory deliverable, or any multi-stream effort, the gaps become serious risks. The question to ask is not "do we have a document?" but "has every planning decision been made and agreed somewhere?"
Q9: At release time the product owner asks to risk-accept two open P2 defects and ship the CoverNZ claims portal anyway. Your exit criteria say "all P2 defects resolved or formally risk-accepted with documented justification." What do you do, and why does this wording matter?
The wording "formally risk-accepted with documented justification" means the release can proceed — but only after the product owner records the decision in writing (a Jira comment, email, or sign-off field) naming the defects, the assessed risk, and their authority to accept it. This is not a blocker; it is a structured decision. The reason the phrasing matters is that it shifts accountability to the right person and creates an auditable trail. In a regulated context like CoverNZ (handling sensitive personal and health data), undocumented risk-acceptance decisions are an audit and liability exposure. Without this clause in the exit criteria, the same decision would happen informally — and the tester would have no record of having flagged it.
★ Interview Questions
What NZ hiring managers ask about Test Planning — and what strong answers look like at each level.
Q: What is the difference between a test strategy and a test plan?
Strong answer: A test strategy is an organisational document that defines the standard approach to testing across all projects — which levels are performed, which techniques are preferred, and how regression is managed. A test plan is project-specific: it derives from the strategy but defines scope, schedule, resources, risks, and entry/exit criteria for one particular release. In practice many agile teams keep the strategy in their working agreements and the test plan in Confluence, but both need to exist somewhere.
Grad / Junior
Q: Why do we define entry criteria in a test plan, and what happens when teams skip them?
Strong answer: Entry criteria are the pre-conditions that must be met before testing begins — environment stable, code merged, test data seeded, acceptance criteria signed off. Without them, testing starts against broken builds and the results cannot be trusted. I’ve seen teams at NZ government projects begin execution before an environment was stable, spend two days logging defects that were actually environment issues, and then have to re-execute everything once the environment was fixed — doubling the effort.
Grad / Junior
Q: You’re testing an Revenue NZ myIR portal release. Requirements are still being finalised two days before sprint testing is due to start. Walk me through how you handle that.
Strong answer: Incomplete acceptance criteria is a direct violation of my entry criteria, so I would not start execution. I’d raise it as a plan risk with a named owner (the product owner) and a deadline, document it in writing — a Jira comment or email is enough — and use the two days for test design and review rather than execution. In an Revenue NZ context you also need an audit trail: undocumented scope assumptions create a compliance exposure if a defect surfaces post-release. I’d make it clear to the project manager that the execution start date needs to shift or the exit criteria will be at risk.
Senior
Q: A release deadline arrives and two P2 defects are still open in the CoverNZ claims portal. The product owner wants to ship anyway. Your exit criteria say “all P2s resolved or formally risk-accepted with documented justification.” What do you do?
Strong answer: The exit criteria already allow for this — “formally risk-accepted with documented justification” is the key phrase. I’d ask the product owner to record the decision in writing: a Jira comment naming the defects, the assessed risk, and their authority to accept it. The release can proceed once that’s done. What I would not do is let the decision happen verbally with no record. CoverNZ handles sensitive health and personal data under the Privacy Act 2020 — undocumented risk acceptance is a liability and audit exposure. Good exit criteria are not a veto; they are a structured decision framework that puts accountability in the right place.
Senior
Q: When would you advise against writing a formal test plan? What’s the trade-off?
Strong answer: I’d skip a formal plan for a two-day hotfix where scope is agreed and acceptance criteria are clear — the AC is the plan, and adding a document adds no value. I’d also skip it when the team has mature working agreements, strong per-story AC, and a Definition of Done that genuinely covers all the planning decisions. The trade-off is visibility: a plan is also a communication and audit artefact. If you’re coordinating across multiple testers, handing work to a contractor, or delivering into a regulated environment like health.govt.nz systems or Benefits NZ benefit payments, the lack of a documented plan becomes a risk in itself. The test is: has every planning decision been made and agreed somewhere — not whether a document exists.
Senior
Q: How do you help a team that writes test plans but never actually uses them — the plan is filed after the kick-off meeting and ignored for the rest of the sprint?
Strong answer: This is the most common test-planning failure I see in NZ teams. I’d start by reducing the plan to the elements the team actually needs — scope, entry/exit criteria, top three risks — and putting it somewhere visible, like a Confluence page pinned to the sprint board rather than a Word document in SharePoint. Then I’d introduce a ten-minute plan review at sprint planning: did scope change? Did any risks materialise? Are entry criteria still valid? The goal is to make updating the plan cheaper than ignoring it. I’d also tie exit criteria directly to the sprint’s Definition of Done so the plan is referenced at every sprint review, not just at kick-off. Measurement: the proxy for a plan being used is whether exit criteria drive the release conversation — if someone at release asks “are we done?” and nobody opens the plan, it’s theatre.
Lead
◆ What I would do
Professional judgment — when to reach for Test Planning, when to skip it, and what to watch for.
If…
Testing a Pharmac drug-dispensing rules update with three testers, a shared environment, and a hard regulatory go-live date
I would…
Write a lightweight plan on day 1 covering explicit scope-in and scope-out, entry criteria that include sign-off on the dispensing rules before execution starts, measurable exit criteria with a 100% critical-path pass requirement, and a risk register with environment instability as Risk 1. Then schedule a ten-minute plan review each Monday. In a health-sector context the plan is also the audit artefact — it cannot be retrospective.
If…
A developer asks whether testing can start today even though three acceptance criteria are still marked as “in review” by the product owner
I would…
Say no to execution, but yes to test design. Use the time to write test cases for the features whose AC is final, flag the open items as a plan risk with the product owner as named owner and a 24-hour deadline, and document this in writing — a Jira comment is enough. Starting execution against unfinished AC guarantees rework. Raising it as a plan risk creates accountability without blocking the whole sprint.
If…
Solo tester, two-day sprint to patch a cosmetic bug in a well-understood UI component, full AC agreed upfront
I would…
Skip the formal plan entirely. The AC is the scope, the Definition of Done is the exit criterion, and the fix is not touching any integration points. Writing a plan here is overhead that slows delivery without adding coverage. The planning content exists — I just do not need a separate document to hold it.
The bottom line: A test plan is not a document — it is a set of decisions that must exist somewhere; the only question is whether they are explicit and agreed before testing starts, or implicit and discovered after a defect ships.
Key takeaway
A test plan is not the document — it is the decisions; the document is just the proof that they were made before testing started, not discovered after a defect shipped.
Related techniques
Test planning is the home for risk-based decisions. Use Risk-Based Testing to populate the risk register section of your plan and to determine which features get the most test depth.
Exit criteria should be expressed as metrics thresholds. Test Metrics & Reporting provides the KPIs to use as the specific numbers in your exit criteria.
The estimation section of a test plan draws on Test Estimation techniques — WBS, three-point estimation, and historical metrics.
Practice this technique: Try Test Lead Practice 01 — Test strategy review.