Regression Strategy
The 5%-rate defect and the keyboard defect are fixed in build 1.1. Release is a few days away. You cannot re-run the entire suite in the time you have — so you build a risk-based regression plan that protects the member’s money without blowing the date.
1 The Hook
The developers have fixed both defects you raised: the contribution-rate validation now restricts to 3, 4, 6, 8 and 10%, and the rate selector is reachable by keyboard. Build 1.1 is in the test environment. Release to members is scheduled for Friday. It is Wednesday.
You have a full test suite covering login, balance, contributions, contact details, and more — days of testing. You have until Friday. Two paths open up. One: re-run everything, miss the date, and frustrate a board that funded this rebuild. Two: just re-test the two fixed defects, confirm they are gone, and ship — fast, but blind to whether the fix broke something else.
Both are wrong. The validation fix touched the contributions code, which shares logic with how rates are sent to the fund administration system — the same path the balance display and the audit log depend on. A fix is a change, and a change can break things far from where it was made. But re-testing everything to be safe ignores that most of the system was not touched at all. The skill this week is choosing the few areas where this change could plausibly have caused damage, and re-testing those deliberately — risk-based regression. It is how real teams ship safely against a real deadline.
2 The Rule
Every fix is a change, and every change can break something it did not touch. You can rarely re-test everything, so re-test by risk: the area that changed, everything that depends on it, and the parts of the system where a failure would hurt the most. Regression is not re-running the whole suite — it is choosing what could have broken and what you cannot afford to ship broken.
3 The Analogy
A surgeon who tugs one stitch and checks the whole wound.
When a surgeon adjusts one stitch, they do not re-open the entire surgery, and they do not just tie off the one stitch and walk away. They check that stitch, then the tissue around it that the adjustment could have pulled on, and they pay closest attention to anywhere a failure would be life-threatening. They re-check by proximity and by consequence.
That is regression selection. The fix is the stitch. You re-test the change itself, then what is connected to it — the contributions path, the fund administration call, the audit log — and you weight your attention toward the things that hurt members most if they fail. Re-running the whole suite is re-opening the surgery; testing only the fix is walking away after one stitch.
4 What Regression Testing Actually Is
Regression testing checks that a change has not broken something that previously worked. It is different from confirmation testing, and the two are often run together but mean different things:
- Confirmation (re-testing): proving the specific defect is actually fixed. Re-run TC-103-07 — does the portal now reject 5%? This verifies the fix did what it claimed.
- Regression: proving the fix did not break anything else that used to work — the surrounding behaviour, the things downstream, the features that share code with what changed.
You always do both. Confirming the fix without regression risks shipping a build where 5% is now rejected but the audit log silently stopped recording changes. Doing regression without confirmation risks declaring victory on a bug that was never actually fixed. This week is mostly about the regression half, because that is the harder judgement call.
5 Risk-Based Selection
You cannot run every test, so you rank them by risk and run the top of the list. Risk here is the familiar combination: how likely is this area to have been affected, and how bad is it if it fails. For the Tūāpapa release, the regression candidates rank roughly like this:
Medium risk — sample: balance display (reads from the same administration system), and the keyboard/screen-reader behaviour of nearby controls (the accessibility fix may have shifted focus order elsewhere).
Lowest risk — skip or smoke only: contact-details update and login, which share no code with the change and were green last run.
Notice what drives the ranking: not how important the feature feels in general, but how plausibly this particular change could have damaged it, weighted by the harm if it did. Login is important, but nothing in this fix went near it.
6 Change-Impact Analysis
To rank by risk you have to understand what the change actually touched. That is change-impact analysis, and you do it by asking the team and reading the change, not by guessing from the UI.
- What code changed? Ask the developer which files or modules the fix touched. The rate-validation fix may live in shared input-handling code used by other forms.
- What depends on it? Trace downstream — the fund administration call, the audit log, anything that consumes the rate. A change to a shared function ripples.
- What is nearby in the UI? The accessibility fix changed focus order; controls next to the selector may now behave differently.
- What is the blast radius if wrong? A broken audit log is invisible to a member but a serious problem for the FMA. Weight the consequence, not just the likelihood.
The defects you logged in Week 4 already point at the highest-risk areas, because they tell you exactly what was changed and why. Your earlier artefacts feed this plan — that is the capstone connecting up.
7 Building the Regression Plan
A regression plan is a short, defensible document: what changed, what you will re-test and why, what you will skip and why, and what evidence you will produce. Here is the shape for the build 1.1 release:
Change: Fixed rate validation (now 3/4/6/8/10% only) and keyboard access to selector.
Re-test (confirm fixes):
• TC-103-07 (5% rejected), TC-103-11 (selector keyboard-reachable + screen reader).
Regression (high risk):
• Full contribution-rate suite (valid, invalid, boundaries).
• Rate sent to fund administration system on a valid change.
• Audit log records old/new rate, member, timestamp.
Regression (medium — sample):
• Balance display reads correctly. Tab order on the Contributions page overall.
Skip (justified):
• Login, contact details — no shared code with the change; passed in build 1.0.
Evidence: run log per case, build 1.1, audit-log rows for the fund-admin and audit checks.
This plan can be shown to Aroha and defended to an auditor. It says what you tested, what you did not, and why each decision was safe — which is exactly what next week’s release recommendation is built on.
8 Common Mistakes
🚫 Only re-testing the fix itself
Why it happens: The bug is fixed, the re-test passes, and it feels like the job is done.
The fix: Confirming the fix proves the bug is gone, not that the fix broke nothing else. A change ripples through shared code and downstream systems — re-test the contributions path, the fund administration call, and the audit log, not just the one failed case.
🚫 Insisting on re-running everything “to be safe”
Why it happens: Full coverage feels safest and avoids having to justify skipping anything.
The fix: Re-running untouched areas burns the time you need for the risky ones and usually misses the date. Spend your budget where the change could plausibly have caused harm; skip what it demonstrably did not touch, with a one-line reason for each skip.
🚫 Ranking by how important a feature feels, not by what changed
Why it happens: Login and balance feel like the “important” features, so they get re-tested by reflex.
The fix: Regression risk is about whether this change could have affected the area, weighted by the harm if it did — not the feature’s general importance. Login matters, but if nothing in the fix went near it, it is low regression risk this release.
🚫 Skipping the invisible downstream effects
Why it happens: The audit log and the fund administration call are not on screen, so they fall off the regression list.
The fix: The most dangerous regressions are silent — a fix that quietly stops the audit log recording, or sends a wrong value to the administration system. These have high blast radius for a regulated product, so they belong at the top of the plan, not off it.
9 Now You Try
Three graded exercises on regression for the Tūāpapa release. Write your answer, run it for feedback from your test lead, then compare to the model answer.
A teammate proposes the regression scope below for build 1.1. Identify 3 problems with it — areas wrongly skipped, areas wrongly included, or missing reasoning — and explain each.
“The fix was just the rate validation, so I’ll re-test that one screen. I’ll also re-run the whole login suite because login is the most important feature. I won’t bother with the audit log or the fund administration call since the member can’t see those. Should be quick.”
List 3 problems and explain each:
Show model answer
This scope re-tests the wrong things. Any three well-explained earn full marks. 1. Skipping the audit log and fund administration call is the biggest error — these share the changed contributions path and are exactly where a silent regression would hide. They are invisible to the member but high blast radius for the FMA, so they belong at the TOP of the plan, not off it. 2. Re-running the whole login suite is misplaced effort — login is "important" in general but shares no code with the rate-validation change, so it is low regression risk this release. That time is being spent on something the change couldn't have touched, instead of on the contributions downstream. 3. Re-testing "that one screen" only is too narrow — it confirms the fix but not the regression. The full contribution-rate suite (valid, invalid, boundaries) and the downstream effects need re-testing, not just the single fixed case. There is also no justification given for any skip. The lesson: rank by what this change could have affected weighted by harm, not by which feature feels important, and never drop the invisible downstream checks.
Below are six areas of the Tūāpapa portal. For build 1.1 (rate-validation fix + keyboard-access fix), rank each as High, Medium, or Skip for regression, and give a one-line reason. Make your reasoning about what changed, not general importance.
Rank each and justify:
Show model answer
(a) contribution-rate flow — High — this is the code that changed; re-test valid, invalid and boundary cases in full. (b) audit log of rate changes — High — depends on the changed contributions path and a silent failure here is a serious FMA concern; high blast radius. (c) RealMe login — Skip (smoke only) — shares no code with the change and passed in build 1.0; re-running the full login suite is misplaced effort. (d) balance display — Medium — reads from the same fund administration system the rate change writes to, so a shared-path regression is plausible; sample it. (e) rate sent to fund administration system — High — shares the changed path and carries the member's actual contribution; a wrong value here directly affects their savings. (f) contact-details update — Skip — separate code from the change, green last run; one-line justification for skipping. The key: (a), (b) and (e) are High because the change touches or feeds them; (c) and (f) are Skip because nothing in the fix went near them — importance in general is not the test.
Write a complete risk-based regression plan for the build 1.1 release. Include: the change summary, the confirmation (re-test) cases, the high-risk regression items, the medium-risk sample, the justified skips, and the evidence you will produce.
Show model answer
Regression plan — Tūāpapa portal build 1.1 Change: Contribution-rate validation now restricts to 3/4/6/8/10% only; the rate selector is now reachable and operable by keyboard. Re-test (confirm fixes): TC-103-07 — 5% is now rejected with allowed rates shown; TC-103-11 — selector reachable by keyboard and announced by a screen reader. Regression — high risk: full contribution-rate suite (valid rates, invalid in-range like 5%/7%, boundaries 2%/3%/10%/11%); the rate is correctly sent to the fund administration system on a valid change; the audit log records member ID, old rate, new rate and timestamp on every change. Regression — medium (sample): balance display still reads correctly from the fund administration system; overall keyboard/Tab order on the Contributions page (the accessibility fix may have shifted focus elsewhere). Skip (with justification): RealMe login — shares no code with the change, passed in build 1.0 (smoke check only); contact-details update — separate code path, green last run. Evidence to produce: run log per case against build 1.1; screenshots/audit-log rows for the fund administration and audit-log checks; a short note recording the skip justifications for the release report. A strong plan re-tests the fix AND the downstream path, weights the invisible audit/fund-admin checks as high, justifies every skip in one line, and names the evidence — which is exactly what Week 6's go/no-go decision will rest on.
10 Self-Check
Click each question to reveal the answer.
Q1: What is the difference between confirmation testing and regression testing?
Confirmation (re-testing) proves the specific defect is actually fixed — re-run the failed case and check the bug is gone. Regression proves the fix did not break anything else that used to work. You do both: one without the other ships either an unfixed bug or a fix that quietly broke something nearby.
Q2: Why is re-running the entire test suite usually the wrong answer?
Because it burns the limited time you have on areas the change never touched and usually misses the release date, while adding little safety. Most of the system was not changed. Spend the budget where the change could plausibly have caused harm, and justify skipping the rest.
Q3: What two factors decide how high-risk an area is for regression?
How plausibly this particular change could have affected the area (proximity — shared code, downstream dependency, nearby UI), and how bad it would be if that area failed (blast radius). High likelihood and high harm puts an area at the top of the plan.
Q4: Why do invisible downstream effects like the audit log belong near the top of the plan?
Because the most dangerous regressions are silent — a fix that quietly stops the audit log recording or sends a wrong value to the fund administration system shows nothing on screen but has high blast radius for a regulated product. A member can’t see them, but the FMA can ask for them.
Q5: What makes a skip decision defensible?
A one-line reason tied to the change, not a shrug — “login shares no code with the contributions fix and passed in build 1.0.” A regression plan is an argument about what could and could not have broken; the justification for each skip is what protects you and the project if something slips.
11 Interview Prep
Real questions asked in NZ QA interviews. Read the model answers, then practise your own version.
“You’ve got a fix going out tomorrow and no time to re-run everything. What do you test?”
I run a risk-based regression. First I confirm the fix actually works by re-running the failed case. Then I do change-impact analysis — I ask the developer what code changed, trace what depends on it, and look at what’s nearby in the UI. I re-test the changed area in full, then everything downstream of it, weighting toward anything where a failure would hurt members or a regulator — on the Tūāpapa portal that’s the fund administration call and the audit log. I skip areas the change demonstrably didn’t touch, and I write a one-line justification for each skip so the plan is defensible.
“A change was made to one screen. Why might a completely different screen break?”
Because changes ripple through shared code and shared data. A rate-validation fix might live in input-handling code used by other forms, or change a value that the balance display and the audit log both read downstream. The accessibility fix changed focus order, which can shift the Tab behaviour of controls next to the one that was fixed. That’s the whole reason regression exists — the danger isn’t where you changed the code, it’s everything connected to it that you didn’t.
“How do you decide what’s safe to skip in regression?”
I skip an area only when I can say in one sentence why this change couldn’t have affected it — it shares no code or data with the change and it passed in the last build. I treat the regression plan as an argument I’d have to defend to an auditor: here’s what changed, here’s what I re-tested and why, here’s what I skipped and why it was safe. If I can’t justify a skip cleanly, that’s a signal it probably isn’t safe and should go back into scope.