Regression Testing
Every change to working software is a risk. Regression testing confirms that changes haven’t broken what was already working. It’s the safety net that enables teams to move fast with confidence.
1 The Hook
A Dunedin fintech ships a small, “obviously safe” change: they tidy up how dates are formatted on a customer statement. The developer tests the statement screen, it looks perfect, and it goes out on a Friday afternoon.
Over the weekend the nightly direct-debit batch fails silently. The same date-formatting helper that drew the statement was also used to build the bank file, and the new format broke the parser at the bank’s end. Nobody re-ran the payments tests, because “we only touched the statement page.” On Monday hundreds of customers had missed payments and a pile of dishonour fees. The change worked exactly as intended — and quietly broke something nobody thought to re-check.
This is the pattern behind regression defects: the new thing works, but a shared component, side effect, or hidden dependency breaks something that used to work. A change is only safe once you have re-checked what it could have affected — not just the part you changed.
Regression testing re-runs previously passing tests after any change to confirm that nothing which used to work has silently broken — it is not about testing the new thing, it is about protecting everything else. Trigger it whenever a change touches a shared component, follows a defect fix, or involves an internal refactor, because the blast radius of a change — not the size of the change — determines what you must re-test. The most common mistake is treating the defect retest as sufficient: confirming a bug is fixed only proves that one thing; regression checks that fixing it did not break something adjacent.
From the field
A NZ insurance platform ran 1,400 regression test cases before every release — each run took 14 days. The team was always behind. They cut to 340 risk-prioritised scenarios covering 85% of claim revenue paths and automated 180 of them. Release cadence went from quarterly to fortnightly. The full suite still exists for major releases — but the team stopped running it for every change the day they admitted they never had time to anyway.
The most dangerous regression failures I’ve seen were not caused by large changes — they were caused by a one-line change to a shared helper nobody thought to trace. Developers confidently scope regression to what they touched. What they miss is the blast radius: every caller, every consumer, every downstream system that silently depends on that code. In NZ government and banking projects I’ve seen a date-format tweak break overnight batch files, and a GST rounding fix corrupt a month of general-ledger reconciliations. Your job is not to trust the developer’s scope estimate — it is to ask “what else calls this?” before agreeing to any regression scope, every single time.
2 The Rule
Every change can break something that already worked, so after any change you must re-run a set of previously passing tests — scoped to the change’s blast radius — to confirm nothing regressed, not just that the new thing works.
3 The Analogy
Renovating one room in an old villa.
You renovate the kitchen of a 1920s Kiwi villa. The new kitchen looks great — but a sensible builder then checks the rest of the house, because the wiring and plumbing are shared. Did the new oven trip the old fuse box? Did moving the sink drop the water pressure in the bathroom? The renovation “works”, but a good builder confirms the lights still come on in the lounge and the shower still runs hot before calling it done.
Regression testing is walking the rest of the house after the reno. You do not re-test every room every time — you check the ones on the same circuit as what you changed. A full rewire warrants checking the whole house; swapping a tap warrants a quick look at the rooms downstream.
Common Mistake vs What Works
The regression suite runs everything every sprint. It takes four hours, results are glanced at rather than read, and flaky tests get quietly skipped rather than fixed — because with 2,000 tests, most are green and nobody wants to block a release over something that “usually passes.” Over time the suite becomes background noise the team doesn’t trust.
Tier the suite: smoke runs on every commit (under 5 minutes), targeted regression runs nightly (around 60 minutes), and the full suite runs only pre-release or after a high-blast-radius change. Fix flaky tests immediately — each one erodes trust in the entire suite. A stable 1,800-test suite that the team acts on is worth more than a 2,000-test suite with 200 tests the team has learned to ignore.
What it is
Regression testing re-executes previously passing tests to verify that new code changes haven’t introduced regressions — bugs in functionality that was already working. It’s run after every change: bug fixes, new features, refactors, infrastructure updates.
The challenge: the regression suite grows with every release, but testing time stays fixed. This forces decisions about which tests to run.
Selection strategies
- Full regression — run everything. Accurate but slow. Used for major releases or when change scope is large.
- Risk-based selection — run tests for the changed areas plus any areas they interact with. Balances coverage and speed.
- Change-impact analysis — trace which code changed and which tests cover that code. Run only those tests. Requires good traceability.
- Core / smoke regression — run the most critical tests only. Used for quick confidence checks after small changes.
In practice: most teams run a tiered regression suite — smoke tests on every PR, a targeted suite nightly, and a full regression weekly or before release. The tiers are defined by the test lead based on risk and run time.
Automation and regression
Regression testing is the strongest argument for test automation. Manual regression on a large suite is unsustainable — it’s repetitive, error-prone, and gets skipped under time pressure.
Good automation candidates for regression: stable features, critical paths, high-risk areas, tests that run on every build. Poor candidates: UI tests that change frequently, exploratory or investigation testing.
Where Regression Testing Lives in CI/CD
The regression suite is the gate before production. Its job is not to find new bugs — it is to catch regressions. Every test in it should have a corresponding user story or past defect as its reason for existing.
Maintaining the regression suite
A regression suite that’s never pruned becomes a liability. Signs it needs attention:
- Flaky tests (intermittent failures) erode confidence in the whole suite
- Tests for deleted features still run and waste time
- The suite takes so long to run that teams skip it
- Duplicate tests cover the same path
Review and prune the regression suite at least quarterly. Delete tests that no longer add value; fix or quarantine flaky tests.
Regression vs confirmation testing
- Confirmation testing (retest) — verify a specific defect has been fixed. Directly re-run the test case that failed.
- Regression testing — verify nothing else broke as a side effect of the fix. Broader in scope.
Every defect fix should trigger both: a retest of the original bug, and a regression of the surrounding area.
Practice this technique: Try Senior Practice 09 — Cross-level regression.
4 Industry Reality
- Regression suites in real NZ teams are almost never perfectly maintained. You’ll inherit hundreds of tests — some of which have been failing intermittently for years, some of which cover features that were removed two product cycles ago. Your first job is triage, not blind execution.
- The blast radius is almost always underestimated. Developers will say “it’s just a one-line change” — and technically they’re right. But that one line touched a shared helper used in eight places, three of which nobody thought about. Senior testers push back and trace dependencies before agreeing to a scope.
- Time pressure is constant. Sprints end on Fridays, releases happen Monday morning, and the full regression takes six hours. In practice you are always negotiating: which tier do we run given this change set and this deadline? You need a defensible, pre-agreed tiered suite so the answer isn’t made up on the spot under pressure.
- Flaky tests are political. Quarantining a flaky payment test sounds sensible, but someone will say “so you’re saying we just stop testing payments?” Senior testers document the quarantine decision, assign an owner to fix it, and set a deadline — they don’t silently skip it and they don’t let it block releases forever either.
- Many NZ companies still have large legacy systems (old government portals, finance platforms, utility billing engines) with no test coverage at all. Regression there means manually running through printed test scripts. Getting automation in is a multi-quarter project, not a sprint task. Knowing how to do manual regression well is still a real skill.
5 When to Use It — and When Not To
✓ Use it when
- Any code change touches a shared component, helper, or library used in more than one place — trace every consumer
- A defect has just been fixed — run regression on the area surrounding the fix, not just the retest of the specific bug
- A major refactor happened internally with no visible feature change — internal rewrites are often higher-risk than new features
- A new feature integrates with existing functionality (payments, authentication, data exports) — the integration point is where regressions hide
- Before every production release, regardless of how small the change set seems — the cost of a production regression is always higher than the cost of another test run
✗ Skip (or thin) it when
- The change is genuinely isolated static content (a text label, a typo in a tooltip, a static image swap) with no shared code touched — smoke only is enough
- You’re early in a feature spike or prototype that will be thrown away — don’t build or run a regression suite for throwaway code
- The same area was fully regression-tested in the last 24 hours with no further changes — re-running the identical suite adds no new information
- The deployment is to a development environment only and there is a proper gate before staging — save the full regression for the staging gate
- You have a flaky test on a low-risk area and the cost of fixing it this sprint is disproportionate — quarantine it formally, document it, and fix it next sprint; do not let it block a green suite indefinitely
Context guide
How the right level of regression testing effort changes based on project context.
| Context | Priority | Why |
|---|---|---|
| Revenue NZ or Benefits NZ benefit/tax platform — fortnightly sprint releases | Essential | A broken calculation in myIR or the Benefits NZ Benefit Management System can incorrectly pay or deny thousands of claimants before anyone notices. Regression protects shared calculation engines and downstream integrations every single release. |
| KiwiSaver or banking app (Harbour Bank / KiwiFirst Bank) — continuous deployment pipeline | Essential | Financial transactions must balance to the cent on every deploy. The FMA and Privacy Act 2020 create audit obligations that make silent regressions in transfer or statement logic a compliance — not just a quality — failure. |
| TransitNZ (TransitNZ) licensing or tolling system — scheduled quarterly release | High | Driver licence and WoF integration feeds multiple police and border systems. A regression breaking the licence lookup API would cascade into enforcement errors; quarterly cadence gives time for a full suite but it must not be skipped. |
| Pacific Air booking or loyalty platform — high-frequency patching | High | Seat allocation, Airpoints calculations, and payment flows are tightly coupled. A patch to the fare-rules engine can silently break loyalty point accrual or seat-upgrade logic; automated smoke + targeted regression catches this before customers see it. |
| Council rates or permit portal (e.g. Auckland Council, Wellington City Council) — low release frequency | Medium | Releases happen 2–3 times per year; a targeted regression covering payment, PDF generation, and GIS lookups is sufficient. Full-suite automation ROI is low unless the team is also maintaining the codebase actively. |
| Internal NZ Police or FamiliesNZ case-management tool — isolated single-team module | Low | A standalone reporting module with no downstream integrations and rare change requests does not justify the maintenance cost of a regression suite; a manual smoke test at each release is proportionate until the module gains more coupling. |
Trade-offs
What you gain and what you give up when you choose regression testing.
| Advantage | Disadvantage | Use instead when… |
|---|---|---|
| Catches regressions that developers and manual testers miss — especially in shared components and downstream integrations where the blast radius of a change is invisible to the person who made it. | Suite maintenance is a permanent cost. An untended regression suite accumulates flaky tests, obsolete scenarios, and duplicates until it becomes too slow to run — at which point teams bypass it, eliminating the protection entirely. | The change is purely isolated static content (a tooltip text update, a static image swap) with no shared code touched — smoke testing is proportionate and faster. |
| Provides an audit trail for regulated NZ environments. FMA, HealthNZ, and DIA auditors can ask what was re-tested before a production release; a timestamped regression report is your evidence that due diligence was exercised. | Full regression on every change slows the pipeline. Teams that run 1,400 tests before every deploy find release cadence drops to quarterly — which concentrates risk rather than distributing it. Over-running regression defeats its own purpose. | The same area was fully regression-tested in the last 24 hours with no subsequent code changes — re-running the identical suite adds no new information and only costs time. |
| Automated regression frees testers from repetitive manual re-checking and redirects their attention to exploratory testing, risk analysis, and the kinds of judgement work that automation cannot replicate — giving the team more coverage in the same sprint. | Regression testing finds only regressions — it does not discover new defects in new functionality, usability issues, or emergent behaviour outside the scripted paths. A team that relies solely on regression misses the class of defects that only exploratory testing surfaces. | The codebase is a short-lived prototype or spike that will be discarded — building and running a regression suite for throwaway code wastes time that should go into exploratory testing. |
| A tiered regression strategy (smoke on every PR, targeted nightly, full pre-release) enables continuous deployment without sacrificing confidence — exactly the model used by NZ fintech and government-digital teams shipping multiple times per week. | Regression scope decisions require judgement that is routinely underestimated. Developers naturally understate blast radius; testers who accept the developer’s scope estimate without tracing shared dependencies will consistently under-regress the highest-risk changes. | The system has no existing test baseline (a legacy FamiliesNZ or council platform with zero automated coverage) — invest in building an exploratory charter and a smoke baseline before attempting to define a regression suite. |
Enterprise reality
How regression testing changes at 200–300-developer scale in NZ enterprise
- Manual regression drops to near zero — at organisations like Revenue NZ and Revenue NZ's transformation programme, any suite touching core tax calculations runs nightly in CI with Selenium Grid or Playwright on cloud-hosted agents; a human never clicks through those flows again unless automation breaks.
- Compliance mandates specific regression evidence: systems holding personal data under the Privacy Act 2020 must demonstrate that a release did not alter data handling behaviour, and HISF-aligned health platforms (HealthNZ) require regression sign-off as a formal artefact before production deployment.
- Tooling scales up: CloudBooks and ListRight run regression suites managed in Zephyr Scale or Xray for Jira, with test-impact analysis (via Launchable or built-in Gradle tooling) to run only the subset of tests affected by a given changeset — cutting 4-hour suites to under 20 minutes.
- Cross-squad coordination becomes its own discipline — with 10+ squads sharing a platform (common in TeleNZ or Pacific Bank core banking), a regression failure in a shared payments library can block releases for three or four unrelated teams simultaneously; enterprise QA chapters own a "regression triage SLA" specifying who investigates and resolves within 2 hours to avoid deployment pile-ups.
◆ What I would do
Professional judgment — when to reach for regression testing, when to skip it, and what to watch for.
The bottom line: Regression testing is not about running everything — it is about running the right tests fast enough that the team never feels the need to skip them. A suite that gets bypassed protects nobody; a scoped, tiered suite that runs on every deploy protects everyone.
6 Best Practices
- ✓ Define your tiers before the release pressure hits. Agree with the team: what runs on every PR (smoke), what runs nightly (targeted), what runs before release (full). Write it down. The decision should never be improvised under a deadline.
- ✓ Trace the blast radius before agreeing to a scope. When a developer says “I only changed X”, ask: what else calls X, imports X, or depends on X? Map it before scoping the regression, not after a defect escapes.
- ✓ Run both confirmation and regression after every defect fix. The retest proves the bug is gone. The regression proves nothing adjacent broke. Doing only the retest is a common and costly shortcut.
- ✓ Treat flaky tests as a priority maintenance item, not an inconvenience. A flaky test on a critical path (payments, authentication) is more dangerous than no test at all, because it trains the team to ignore red.
- ✓ Prune the suite on a calendar, not just when it hurts. Schedule a quarterly suite review. Delete tests for retired features. De-duplicate tests covering the same path. An unpruned suite that takes too long gets skipped — which defeats the entire purpose.
- ✓ Automate the repetitive, protect the exploratory. Stable critical paths, payment flows, and core integrations belong in automated regression. Frequently changing UI and exploratory investigation do not — the maintenance cost of brittle automation outweighs its value.
- ✓ Document quarantine decisions formally. If a test is quarantined, write down why, who owns the fix, and when it must be resolved. A quarantine with no owner and no deadline becomes a permanent skip.
- ✓ Scale regression to the size of the blast radius, not the size of the change. A one-line change to a shared core module may warrant full regression. A 200-line change to an isolated feature may warrant only targeted regression. Scope is about impact, not line count.
- ✓ For NZ financial and government systems, keep a regression evidence log. Regulators (FMA, DIA, HealthNZ) may ask what was re-tested before a release. A timestamped report of which tests ran and passed is your audit trail.
- ✓ Agree on the “exit criteria” for regression before testing starts. What pass rate is acceptable? Are any failures blockers? Get this in writing with the team lead before the sprint ends, not during a go/no-go call on release day.
7 Common Misconceptions
❌ Myth: If no new features were added, there’s nothing to regress-test.
Reality: Internal refactors and “purely technical” changes are often the highest-risk deployments on a team’s calendar. Rewriting a pricing engine, upgrading a database library, or changing a shared date parser touches the same code many features depend on — without a visible surface change to prompt anyone to test broadly. Teams that skip regression on refactors are the ones that take down production on a “safe” deployment.
❌ Myth: Once a bug is retested and confirmed fixed, regression is done.
Reality: Confirmation testing (the retest) only proves the one specific defect is resolved. Regression testing asks a different question: did fixing that bug break anything else? Every defect fix should trigger both. The retest is the minimum; the regression around the fix is where the real safety net sits. Skipping it is how one fix introduces two new bugs.
❌ Myth: More tests in the regression suite means better coverage and a safer release.
Reality: A bloated, slow regression suite that teams skip under time pressure provides zero safety. A tight, trusted, well-maintained suite that teams actually run on every release is far more valuable. Regression effectiveness is measured by the defects it catches before production — not by the number of tests in the folder. Pruning dead tests, fixing flaky ones, and keeping run time reasonable is part of the test lead’s job, not an optional extra.
8 Now You Try
Three graded exercises — spot, fix, then build. Write your answer, run it for AI feedback, then compare to the model answer.
A NZ utility’s self-service portal has an 1,100-test regression suite. A change has come in: the shared currency-formatting helper (used on bills, the dashboard, and the bank-file export) was modified to add a thousands separator. Which regression strategy fits — full, risk-based selection, change-impact, or smoke only — and why? Name the specific areas you would re-test.
Show model answer
Chosen strategy: risk-based selection driven by change-impact analysis (not smoke only, and not necessarily the full 1,100 tests). Why: a change to a SHARED helper has a wide blast radius — every place that uses currency formatting is at risk, not just the one screen the developer was thinking about. Smoke only is too shallow; it would likely miss the bank-file break. Full regression is defensible if traceability is poor, but if you can trace usages you can scope it precisely. Specific areas to re-test: every consumer of the currency helper — bill display, the dashboard totals, and crucially the bank-file export (machine-read formats break easily on a new separator). Also any CSV/PDF outputs and any downstream parser that reads those amounts. The bank-file export is the highest-risk consumer because a human eyeballing a bill would forgive a comma, but a parser will not.
A team lead made the call below for a release. It is wrong on several counts. Rewrite the regression approach with the right strategy for each change and fix the reasoning errors.
“We refactored the whole pricing engine but no new features were added, so smoke tests are enough. We also fixed a typo in a tooltip — run the full 1,100-test suite to be safe. The pricing fix passed its own retest, so we don’t need any regression around it. And we’ll skip the flaky payment tests since they keep failing.”
Rewrite the regression approach:
Show model answer
Pricing engine refactor — needs FULL (or broad risk-based) regression, not smoke. "No new features" does not mean low risk; an internal refactor of a core engine can break behaviour anywhere it is used. This is the highest-risk change in the release. Tooltip typo — needs SMOKE only, not the full 1,100 tests. A static text change has essentially no blast radius. Running the full suite here wastes hours for no risk reduction; the lead has the two changes exactly backwards. "Retest is enough, no regression needed" — wrong. Confirmation testing (retest) only proves the specific bug is fixed. Regression checks nothing ELSE broke as a side effect of the fix. Every fix should trigger both: a retest of the bug and a regression of the surrounding area. Skipping the flaky payment tests — do not just skip them. Flaky tests on a critical path (payments) hide real failures. Quarantine and FIX them, or replace them with reliable tests; silently skipping payment regression is how a real payment defect escapes. Flakiness is a maintenance problem to solve, not a reason to stop testing payments.
You are the test lead for a NZ online grocery service. Design a tiered regression suite: define the tiers (e.g. smoke, targeted, full), say what runs in each tier and when it is triggered, name candidates for automation vs manual, and describe how you would keep the suite healthy over time (flaky tests, retired features, duplicates).
Show model answer
A strong tiered design: Tier 1 — Smoke / core: ~20-40 critical-path tests (log in, add to cart, checkout, payment, place order). Triggered on every PR / every build. Fast, fully automated. Purpose: quick confidence that the app is not fundamentally broken. Tier 2 — Targeted / risk-based: tests for the changed areas plus their interactions and shared components. Triggered nightly and on every change before merge to the release branch. Mostly automated, scoped by change-impact analysis. Tier 3 — Full regression: the entire suite. Triggered weekly and before every production release, plus after any large or high-risk change (e.g. a payment-gateway switch or a core-engine refactor). Automated where stable; some exploratory/manual checks for areas automation cannot cover well. Automation candidates: stable features, critical paths, high-risk areas, anything run on every build. Keep manual: frequently changing UI, exploratory and investigation testing, one-off checks. Keeping it healthy: review and prune at least quarterly. Quarantine and fix flaky tests rather than ignoring them (flakiness on payments is dangerous). Delete tests for retired features and de-duplicate tests covering the same path. The goal is a suite the team trusts and actually runs — an unpruned suite that takes too long gets skipped under pressure, which defeats the point.
Self-Check
Click each question to reveal the answer.
Why teams fail here
- The regression suite grows with every release but nothing is ever removed, making it permanently behind schedule
- Test cases are written without considering maintainability — one UI change breaks hundreds of scripts
- Regression scope is not risk-based, so trivial features get the same coverage as business-critical paths
- Automated regression results are ignored because flaky tests have made the suite untrustworthy
How this has changed
The field moved. Here is how Regression Testing evolved from its origins to current practice.
Regression testing is implicit — before release, test what you fixed and hope nothing else broke. No systematic approach. The first documented use of "regression testing" as a named practice appears in software engineering literature in the 1970s.
Test management tools (Mercury TestDirector, later QC/ALM) create regression test suites as managed artefacts. Full regression runs take days or weeks. The weekend regression run becomes a standard waterfall ritual.
Selenium enables automated browser regression testing. JUnit and xUnit frameworks make unit regression testing continuous. The cost of regression testing drops dramatically — tests that took days to run manually take minutes to run automatically.
Continuous integration makes regression testing continuous. Every commit triggers a regression suite. Test selection and prioritisation become important as suites grow — running 10,000 tests on every commit is unsustainable. Risk-based test selection enters CI practice.
AI tools can predict which tests are most likely to fail based on changed code — enabling intelligent test selection that runs the highest-risk subset first. Visual regression (Applitools, Percy) catches UI regressions that functional tests miss. The challenge is flaky tests — AI analysis can identify and quarantine flaky tests automatically.
Interview Questions
What NZ hiring managers ask about Regression Testing — and what strong answers look like.
Your regression suite takes 45 minutes to run. The team wants to deploy multiple times a day. What do you do?
Strong answer: I restructure the suite into tiers: a fast smoke test (under 5 minutes) that runs on every commit, a focused regression tier (under 15 minutes) that runs before every deployment, and a full regression tier (45 minutes) that runs nightly or on release branches. I also look at parallelisation — running tests in parallel across multiple agents typically cuts runtime by 60-80%. I review the slowest 10% of tests for optimisation opportunities (slow browser tests that could be API tests, database setup that could use fixtures). The goal is a deployment pipeline where feedback comes in under 15 minutes.
Mid/Senior
How do you decide which tests belong in a regression suite versus which to retire?
Strong answer: I retire tests that: duplicate coverage provided by faster tests at a lower level (browser test for behaviour already covered by a unit test), test functionality that no longer exists, have been consistently flaky for more than three runs without a fix, or take more than 60 seconds without testing a high-risk area. I keep tests that: cover high-risk or high-complexity functionality, have historically found production bugs, or cover areas that are frequently modified. I run a regular coverage audit — if a test has not caught a regression in a year, I ask whether it is protecting a real risk or providing false assurance.
Senior/Lead
Q1: What is the difference between regression testing and confirmation (retest)?
Confirmation testing re-runs the specific failed test to verify one defect is now fixed. Regression testing checks that nothing else broke as a side effect of the change — it is broader in scope. Every defect fix should trigger both: a retest of the original bug and a regression of the surrounding area.
Q2: Why is “no new features were added” a poor reason to skip deep regression on a refactor?
Internal changes to a core component can break behaviour anywhere that component is used, even with no visible feature change. A refactor of a shared engine often has a wide blast radius, so it warrants broad or full regression — it can be one of the highest-risk changes in a release precisely because it touches so much.
Q3: How does the size of a change’s blast radius map to the regression strategy you choose?
Isolated static changes (a text/typo fix) need smoke only. A change with a known, bounded blast radius (a new feature on one screen) suits risk-based or change-impact selection — the changed area plus its interactions. Broad or unknown-impact changes (a major release, a core refactor) need full regression.
Q4: What makes a test a good automation candidate for regression, and what makes a poor one?
Good candidates: stable features, critical paths, high-risk areas, and tests that run on every build — the repetitive checks humans skip under pressure. Poor candidates: UI that changes frequently, and exploratory or investigation testing, where the cost of maintaining brittle automation outweighs the benefit.
Q5: Why must a regression suite be pruned, and what are the warning signs it needs attention?
An unpruned suite becomes a liability: it gets so slow teams skip it, which defeats its purpose. Warning signs: flaky tests eroding trust, tests for deleted features still running, duplicate tests covering the same path. Review at least quarterly — delete dead tests and fix or quarantine flaky ones.
Q6: Your team is releasing a change to the shared identity-verification module used by both the KiwiSaver enrolment flow and the RealMe login integration on an Benefits NZ portal. The developer says the change is small. What regression approach do you take, and why?
A: A small change to a shared identity module has a wide blast radius: any flow that calls it is at risk regardless of how many lines changed. Run change-impact analysis to find every consumer — at minimum the KiwiSaver enrolment flow, the RealMe login integration, and any downstream services that rely on a verified identity token. Risk-based selection scoped to those consumers is the right tier; smoke only is too shallow for an authentication-adjacent shared component because a failure there could lock citizens out or create compliance exposure under NZ privacy and identity-assurance rules.
Q7: An Revenue NZ tax-filing portal team runs full regression before every release regardless of change size, and a separate team runs only smoke tests before every release. Neither team has had a production incident in six months. Which team has the better regression strategy, and what information would you need to judge?
A: Neither approach can be judged without knowing the change frequency, suite size, and blast radius of typical changes. Full regression before every release is wasteful if most changes are isolated — it slows down delivery and trains teams to treat it as a checkbox. Smoke-only is dangerously shallow if changes regularly touch shared components. The right answer is a tiered strategy: smoke on every PR, targeted regression scoped to the blast radius of each change, and full regression before major or high-risk releases. Six months without incidents is not evidence the strategy is correct — it may mean no high-risk changes shipped, or that defects escaped undetected.
Q8: What is the key difference between regression testing and risk-based testing, and can you use both at the same time?
A: Regression testing answers the question "did the change break something that already worked?" — it is triggered by change and looks backward at existing functionality. Risk-based testing answers the question "which areas are most likely to fail or most costly if they do?" — it is a prioritisation lens applied across all testing, not just re-runs. They work together: you apply risk-based thinking to decide which parts of the regression suite to run first and how deeply, making your regression more efficient. On an TransitNZ driver-licence portal, for example, risk-based thinking tells you the identity-check and payment flows warrant the deepest regression coverage because failure there is high-consequence, while a low-risk help-text update warrants only smoke.
Q9: A developer tells you: "We don't need regression testing because we have 90% unit-test coverage — if anything broke, the unit tests would have caught it." What is wrong with this reasoning and how do you respond?
A: Unit tests verify individual functions in isolation; they do not catch integration failures, configuration drift, or emergent behaviour when components interact at runtime. A shared date-format helper can pass all its unit tests and still break a bank-file parser when the output format is consumed by an external system — exactly the kind of defect regression testing is designed to catch. High unit-test coverage is valuable but it does not replace regression: you need both. A measured response is to agree that unit tests are a strong foundation, then explain that regression testing covers the layer unit tests cannot — how components interact, how the system behaves end-to-end, and whether a change to one part silently broke another. In regulated NZ environments (CoverNZ, HealthNZ, banking) this distinction also matters for audit evidence.
Try It — Select the right regression strategy
A NZ insurance portal has a regression suite of 800 tests. Four different change scenarios have come in. For each one, choose the most appropriate regression strategy.
| Change scenario | Best regression strategy |
|---|---|
| A typo was fixed in the "Thank you" confirmation email template — no code logic changed | |
| The premium calculation engine was refactored — no new features, but significant internal changes | |
| A new "add vehicle" feature was added to the policy management screen | |
| Major release: payment gateway switched from Stripe to POLi + 40 other features shipped simultaneously |
Answers
| Scenario | Strategy | Why |
|---|---|---|
| Typo in email template | Smoke / core regression only | Zero logic change. Confirm the app still works end-to-end, then check the email text. No deep regression needed. |
| Premium calculation refactor | Full regression (all 800 tests) | Internal changes to the calculation engine can break unexpected areas. The refactor may be "no new features" but it's high-risk — run everything. |
| New "add vehicle" feature | Risk-based selection | Run the new feature tests, plus regression on policy management, premium recalculation, and any shared components. Not the whole suite. |
| Major release — 40+ features | Full regression (all 800 tests) | Large change scope means any area could regress. Full suite is the only defensible choice before this release. |
The decision framework: how much of the system could this change affect? Isolated static changes → smoke only. Known blast radius → risk-based. Unknown or broad changes → full regression.