Defect Management
From the moment you find a bug to the moment it’s confirmed fixed — defect management is the process that keeps bugs tracked, prioritised, and resolved systematically.
1 The Hook
A tester at a Hamilton software house finds a real problem: when a customer pays for an order, the money is debited but the order sometimes never gets created. They fire off a Slack message — “hey, payment thing is broken again” — and move on to the next screen.
Three weeks later it surfaces in a production incident. The developer asks: which payment method? Which browser? What was the order total? Was the customer logged in? Nobody knows. The message had no steps, no environment, no evidence, and was never logged as a defect, so it never reached triage and was never prioritised. A genuine money-losing bug was found and then quietly lost, because finding a defect is worthless if it is not captured, classified, and tracked through to a verified fix.
This is the pattern behind a surprising number of escaped defects: the bug was caught in testing and then leaked out of the process. A defect that lives in a chat message instead of a tracked report is a defect nobody owns — and an unowned defect ships.
Defect management is the discipline of moving every discovered bug from an informal observation into a tracked, classified, and verified-closed state — use it any time real users, financial data, or regulatory obligations (Revenue NZ, RBNZ, Holidays Act) are in scope, because an untracked defect is an unowned one that tends to ship. The most common mistake testers make is conflating severity with priority: severity is your technical assessment of impact (set it honestly), while priority is a business decision owned by product — inflating severity to force urgency corrupts the signal the whole team relies on.
From the field
A NZ government agency tracked bugs in a spreadsheet shared between six testers. Two critical defects were marked fixed, re-tested, and closed — but against the wrong build number. The issues shipped to production. The post-incident review found three people had edited the status column on the same day with no audit trail. Moving to Jira with mandatory build field and transition rules eliminated the class of defect entirely. Process is not bureaucracy — it is what stops the same bug shipping twice.
The defect that kills a project is rarely the one that was missed — it is the one that was found, marked Deferred, and then lost. I have inherited handovers on NZ government projects where the Jira backlog contained 200-plus deferred defects from the previous release, assigned to people who had left, with no reproduction steps and no business sign-off on the deferral decision. Nobody owns a deferred defect after the sprint closes unless you force someone to. Before every release, walk the deferred backlog deliberately: re-triage each item, confirm it is still fit to defer, and get the product owner's explicit written acknowledgement in the ticket. A deferred defect without documented sign-off is not a decision — it is just risk your organisation forgot it was carrying.
2 The Rule
A defect is not managed until it is captured in a tracker with enough detail to reproduce it, classified by severity and priority, and moved through a defined lifecycle to a verified, closed state — finding the bug is only the first step.
3 The Analogy
An CoverNZ claim, not a complaint to a mate.
If you hurt your back, telling a friend “my back’s sore” gets you sympathy and nothing else. Lodging an CoverNZ claim is different: it records what happened, when, and how, gives it a claim number, routes it to the right assessor, tracks it through review and treatment, and closes it only when you are signed off. The structure is what turns “something is wrong” into “something is being dealt with, and we can see where it is up to.”
Defect management is lodging the claim, not moaning to a mate. A good bug report is the claim form — specific enough to act on without a phone call — and the lifecycle states are the claim status, from lodged to assigned to resolved to closed.
Common Mistake vs What Works
The bug is logged as “it didn’t work” with a screenshot but no reproduction steps, the environment field blank, and severity set to Critical because it felt urgent. The developer cannot reproduce it, reopens the ticket asking for more information, and three days pass while the real issue sits unfixed — or quietly disappears from the backlog.
Every bug report includes: the environment (browser, OS, build), numbered steps to reproduce from a known starting state, expected result, actual result with the error message verbatim, and a severity that reflects genuine business impact. If you cannot write reproducible steps, the defect is not ready to log. “Critical” means users cannot complete a primary workflow — not “this annoyed me.” In NZ payroll or compliance contexts, a clear report with accurate severity also creates the paper trail regulators expect.
The defect lifecycle
| State | Who owns it | What happens |
|---|---|---|
| New | Tester | Bug reported, not yet reviewed |
| Assigned | Lead / Manager | Triaged, assigned to developer |
| Open / In Progress | Developer | Being investigated and fixed |
| Fixed | Developer | Code change made, ready for retest |
| Retest | Tester | Verify the fix works |
| Closed | Tester | Verified fixed in all affected environments |
| Rejected | Developer / Lead | Not a bug — by design, duplicate, or not reproducible |
| Deferred | Lead / PM | Won’t fix this release, scheduled for later |
Writing good bug reports
A good bug report lets a developer reproduce the issue without asking you anything. It contains:
- Title — one sentence: what breaks, where, and under what condition. Not "Login broken" — "Login fails with HTTP 500 when password contains special characters".
- Steps to reproduce — numbered, specific, reproducible steps from a known starting state.
- Expected result — what should happen.
- Actual result — what actually happens. Include error messages verbatim.
- Environment — browser, OS, version, test environment.
- Severity and priority — (see below).
- Evidence — screenshot, video, log snippet.
Severity vs priority
These are different dimensions. Both must be set, and they often conflict.
| Severity | Definition | Example |
|---|---|---|
| Critical | System crash, data loss, security breach | Payment processed but order not created |
| High | Major function broken, no workaround | Users can’t submit the main form |
| Medium | Function broken, workaround exists | Export to CSV fails but PDF works |
| Low | Minor issue, cosmetic, or rarely hit | Tooltip text has a typo |
Priority is a business decision: how urgently does this need to be fixed, relative to everything else? A low-severity cosmetic bug on the homepage might be high priority if a CEO demo is tomorrow. A high-severity crash in a rarely used report might be deferred.
Testers set severity; product/business sets priority. This distinction matters — don’t conflate them.
Root cause analysis
For high-severity or recurring defects, root cause analysis (RCA) asks not just "what broke" but "why." Common RCA techniques: 5 Whys, fishbone diagram, fault tree analysis. The goal: prevent the same class of bug recurring. RCA findings feed into process improvements and updated testing checklists.
Defect metrics
- Defect density — defects per unit of code or per feature area. Identifies unstable modules.
- Defect removal efficiency (DRE) — percentage of defects found before release vs total defects found (pre + post release). Target: > 90%.
- Mean time to detect (MTTD) — average time from defect introduction to detection.
- Defect escape rate — how many defects reach production. The key quality signal for leads.
ISTQB mapping
| Ref | Topic | Level |
|---|---|---|
| CTFL Ch. 5 | Defect management — defect reports, lifecycle, workflow | Foundation |
| CTAL-TA Ch. 6 | When defects can be detected, defect report fields, classification | Advanced |
| CTAL-TM Ch. 4 | Defect management process, metrics, root cause analysis | Lead |
Practice this technique: Try Test Lead Practice 08 — Defect triage, Test Lead Practice 09 — Coaching scenario.
4 Industry Reality
- Bug trackers are never clean. Most projects inherit a backlog of hundreds of deferred or unresolved defects in Jira or Azure DevOps — many with no steps to reproduce, wrong severity, or an assignee who left the company two years ago. Your first job is often triage of other people's mess, not new defect logging.
- Developers push back constantly. "Can't reproduce", "works on my machine", "by design" — these are the most common responses in real teams. Senior testers prepare for this: they log with detailed environment info, attach a screen recording, and note the build number so there is no wiggle room. If it can't be reproduced, the defect goes to "Pending info" not "Rejected".
- Severity and priority get misused everywhere. Developers raise everything to Critical to get attention; product managers P1 everything regardless of impact. In practice you will spend significant time coaching stakeholders on the difference, and you may need to escalate to a test lead when priority is being gamed to jump the queue.
- The Holidays Act is a live defect generator in NZ. Payroll systems in NZ have been chronically non-compliant with the Holidays Act 2003 — this has cost major NZ employers (including government departments) tens of millions in back-pay. Defects in payroll, leave, and entitlement systems carry outsized regulatory and financial risk and should default to High severity.
- Deferred defects ship anyway. A defect marked "Deferred — fix next sprint" frequently never gets fixed, especially if the project delivers and the team moves on. Senior testers flag high-risk deferred items explicitly in their test summary reports and ensure product owners sign off in writing, so the risk decision is documented and owned.
5 When to Use It — and When Not To
✓ Use it when
- Any defect is being reported in a product that has external users or downstream dependencies — formal tracking is non-negotiable.
- The system handles financial, health, or safety data (e.g. RBNZ-regulated banking systems, health.govt.nz portals) where an untracked bug could have legal or compliance consequences.
- The team is larger than two people — informal chat-based bug reporting breaks down immediately at team scale.
- You need to demonstrate quality to a third party — a structured defect report is the evidence of due diligence in audits, go-live reviews, and vendor acceptance testing.
- A bug keeps reappearing after being "fixed" — formal lifecycle tracking (and RCA) is the only way to stop the cycle of regressions.
✗ Skip it when
- You are doing rapid exploratory testing on a throw-away prototype or wireframe where the code will be deleted — overhead of formal logging exceeds the value.
- It is a solo developer project with no handoff — a sticky note or inline comment is proportionate; a full Jira workflow is not.
- The "defect" is actually an outstanding design question — log it as a question or spike, not a bug, so it is not sitting in a defect backlog muddying your metrics.
- You are in a pair-programming or mob-testing session where the developer is sitting next to you and the fix takes 30 seconds — just fix it and capture a note in the session log rather than opening a ticket for noise.
Context guide
How the right level of defect management effort changes based on project context.
| Context | Priority | Why |
|---|---|---|
| Revenue NZ, Benefits NZ, or CoverNZ systems handling financial entitlements or benefit payments | Essential | A defect in a payment or entitlement calculation carries direct financial harm and potential Holidays Act or KiwiSaver Act liability. Formal tracking with severity classification and documented deferral sign-off is legally defensible; informal handling is not. |
| HealthNZ or patient-data systems governed by the Health Information Privacy Code | Essential | Privacy and data-integrity defects in health systems require an audit trail from discovery to verified close. The Privacy Act 2020 creates a notifiable-breach obligation, so every defect involving patient data must be tracked and classified before a release decision is made. |
| Harbour Bank, Pacific Bank, or other RBNZ-regulated banking and payments platforms | Essential | RBNZ operational-risk requirements and PCI-DSS compliance mandate that every defect affecting transaction integrity or security controls is captured, classified, and resolved with documented evidence. Escape to production triggers a mandatory incident-reporting process. |
| Pacific Air or TransitNZ systems with safety-adjacent features | High | Safety-critical workflows (crew scheduling, vehicle inspection records) require full defect lifecycle tracking with RCA on any severity-Critical escape. The paper trail is evidence in safety audits and CAA or TransitNZ reviews. |
| Spark, Vodafone, or SaaS platforms with large consumer audiences but no safety or regulatory obligations | Medium | Formal defect tracking in Jira or Azure DevOps with severity and priority is still valuable for escape-rate metrics and regression prevention, but full RCA on every defect is disproportionate. Focus RCA on production escapes and recurring defects. |
| Internal tooling, short-lived prototype, or solo developer project with no handoff | Low | Lightweight tracking (a shared spreadsheet, a GitHub issue) is proportionate. Full lifecycle workflow adds overhead without corresponding value when there is no external user, no compliance obligation, and no handoff requirement. |
Trade-offs
What you gain and what you give up when you choose formal defect management.
| Advantage | Disadvantage | Use instead when… |
|---|---|---|
| Every defect has an owner, a state, and a paper trail — nothing is lost because someone forgot to follow up a Slack message. | Adds process overhead per defect: logging, triaging, assigning, retesting. On a fast-moving solo project, the overhead can exceed the value. | The fix is trivial, the developer is sitting next to you, and the change will be verified in under a minute — a session note is proportionate. |
| Defect metrics (escape rate, DRE, defect density by module) become available, turning gut feel about quality into measurable data. | Metrics are only useful if fields are filled in consistently. A tracker where half the defects have blank severity or no build number produces misleading signals. | The team does not have the discipline to maintain field quality — consider a lighter-weight tool (GitHub Issues, a shared spreadsheet) with fewer mandatory fields before moving to a full lifecycle workflow. |
| Deferral and rejection decisions are documented and owned — the audit trail protects testers and the organisation if a deferred defect later ships to production. | Formal process can create a false sense of control. A well-structured Jira workflow does not compensate for poor test coverage or inadequate environments. | The organisation is in early-stage exploration mode and the product is not yet exposed to real users — invest in test coverage before process overhead. |
| RCA findings feed back into the development process — preventing entire classes of defect rather than patching individual instances. | Full RCA on every defect is expensive in time and team attention. On a project shipping weekly, running a fishbone on every medium-severity bug is unsustainable. | Reserve full RCA for production escapes, severity-Critical defects, and any defect that reappears after a fix — not as a routine step for every low-severity issue. |
Enterprise reality
How defect management changes at 200–300-developer scale in NZ enterprise — where informal tracking fails and the consequences of getting it wrong are regulatory, not just operational.
- What gets automated: At small-team scale, severity classification, duplicate detection, and stale-ticket chasing are manual work in a standup. At enterprise scale they are enforced by the tool. Jira automation rules block a defect from transitioning to “Fixed” without a linked test case and build tag; scripted fields auto-escalate any ticket sitting in “New” for more than 48 hours; duplicate detection plugins surface probable duplicates the moment a reporter saves a ticket. The goal is to eliminate the class of human-error failures that kill defect data quality at volume — blank severity fields, no environment, no assignee — by making them impossible to submit rather than relying on discipline.
- Governance and compliance: Under the Privacy Act 2020, a privacy-class defect that escapes to production triggers a mandatory notifiable-breach notification to the Privacy Commissioner within 72 hours of the organisation becoming aware of it — that clock starts the moment the defect is confirmed, not when it is fixed, so defect classification at triage is a legal act, not a QA process step. KiwiFirst Bank and Coastal Bank must demonstrate end-to-end defect traceability for any change to payment or credit systems under the RBNZ’s Operational Risk Requirements (BS11). NZISM mandates that security defects in government systems are remediated within SLAs tied to their classification. “We fixed it informally” is not an audit response at any of these organisations.
- Tooling at volume: Jira Software with Zephyr Scale or Xray for test linkage is the dominant stack in NZ private sector enterprise (Pacific Air, TeleNZ, CloudBooks, ListRight). Azure DevOps is the default in central government (Revenue NZ, Benefits NZ, TransitNZ) for its native integration with Microsoft compliance tooling and NZISM-aligned cloud environments. The tracker matters less than the configuration: mandatory custom fields, transition guards, CI/CD pipeline integration that stamps every defect with the failing build hash, and automated dashboards surfacing escape rate and ageing deferred tickets to senior stakeholders. At 10+ squads, a poorly configured tracker produces defect data nobody trusts and metrics nobody acts on.
- Cross-squad coordination: Defects that cut across squad boundaries — a shared authentication service, a platform API, a common data pipeline — are the most dangerous class at this scale because no single squad owns them. Without explicit governance, an inter-squad defect sits in the backlog of the squad that found it while every other affected squad assumes someone else is handling it. The consequence at HealthNZ scale is a patient-record defect that surfaces in three separate squad contexts simultaneously, gets fixed three times independently with no shared regression sweep, and re-surfaces in a fourth context the next sprint. The fix is structural: a weekly cross-squad defect triage, a directly responsible individual (DRI) assigned to every shared-component defect, and escalation paths that route inter-squad defects to a platform owner rather than decaying in individual squad backlogs.
◆ What I would do
Professional judgment — when to reach for formal defect management, when to scale it back, and what to watch for.
The bottom line: Defect management is most valuable not when bugs are found, but when they are classified honestly, tracked to verified closure, and deferred or accepted only with explicit documented ownership. The paper trail is the product — it is what protects testers, organisations, and users when something goes wrong.
6 Best Practices
- Write the title as a one-sentence failure statement: [What] fails [where] when [condition]. "Login fails with HTTP 500 when password contains an ampersand" — not "Login broken".
- Always record the build number or deployment tag in the environment field. If you cannot tell the developer exactly which build you tested on, the defect report is worthless the moment a new build is deployed.
- Attach a screen recording as the default, not an afterthought. A 15-second recording is faster to produce than a perfect written reproduction path, and it eliminates "can't reproduce" disputes.
- Set severity based on impact, not on urgency or your feelings about the bug. A critical bug that blocks a payment is Critical. A cosmetic typo on a demo page is still Low severity, even if it is High priority. Resist the temptation to inflate severity to get faster attention.
- Never close a defect yourself if you did not verify it. The tester who found the bug should be the one who verifies the fix — close the ticket only after retesting in the same environment where the bug was found.
- Run a regression sweep around every fix, not just a point retest. Payroll bugs in NZ are a classic example: fixing one entitlement calculation often breaks adjacent leave types. Test the boundary conditions around the fix, not just the exact reproduction steps.
- Push for RCA on any defect that escaped to production, or any defect that reappears after a fix. The goal is not blame — it is process improvement. A defect that ships once is a process gap; a defect that ships twice is a pattern that needs systemic change.
- Keep your defect metrics visible to the team, not just the test lead. Defect density by module, escape rate, and DRE are signals the whole team should be tracking — they shift behaviour and focus development effort on the most fragile areas.
- When a defect is marked "Rejected" as not reproducible, add a note with the exact reproduction conditions you used and mark it "Pending info" rather than accepting the rejection without evidence. This creates an audit trail and prevents genuine bugs from disappearing.
- For compliance-sensitive defects (RBNZ, Privacy Act 2020, Holidays Act), cc the business owner in the defect notification and document their awareness explicitly. This protects both the tester and the organisation if the defect ships deferred.
7 Common Misconceptions
❌ Myth: If the developer says "can't reproduce", the defect should be closed as Rejected.
Reality: "Can't reproduce" is a signal of an information gap, not confirmation the bug doesn't exist. Move the defect to a "Pending info" or "Need more info" state and provide additional context — a screen recording, a different test environment, a precise build number, or a test account the developer can use to reproduce it themselves. Senior testers do not accept Rejected without evidence. A defect that was reproducible for the tester and is now "can't reproduce" has likely masked an intermittent or environment-specific issue.
❌ Myth: The tester who finds the bug decides its priority.
Reality: Testers own severity (the technical impact assessment). Priority is a business decision owned by the product owner, project manager, or business stakeholder — it depends on factors the tester may not have visibility into, such as regulatory deadlines, customer commitments, or release schedules. A tester who unilaterally marks every bug P1 to get it fixed faster is corrupting the prioritisation signal and training the team to ignore priority fields. Set severity accurately; let business set priority.
❌ Myth: Once a defect is marked Fixed and passes retest, it is done.
Reality: A verified fix is only one part of closing a defect properly. You also need to confirm the fix has been deployed to all affected environments (not just the developer's local environment), run a regression check on the adjacent functionality, and for any high-severity defect, document in the tracker that RCA was completed and a process improvement was actioned. Defects that are verified-fixed-but-not-regressed are a common source of escaped defects — the original bug is gone, but the fix broke something else.
8 Now You Try
Three graded exercises — spot, fix, then build. Write your answer, run it for AI feedback, then compare to the model answer.
A defect on a NZ supermarket online-shopping site: “On the checkout page, the ‘Pay now’ button is mislabelled ‘Pat now’. The Pak’nSave board is being shown the live site in a press demo tomorrow morning.” Assign a severity and a priority to this bug, and explain why they differ.
Show model answer
Severity: Low. It is a cosmetic typo — nothing is broken, no data is lost, payment still works. Priority: P1 (Urgent). A board press demo tomorrow makes a visible typo on the checkout page embarrassing and time-critical, so it must be fixed today. Why they differ: severity measures technical impact (how badly is the system broken?), priority measures business urgency (how soon must we fix it relative to everything else?). They are independent dimensions. This is the textbook low-severity / high-priority case: trivial bug, but a business deadline drives the urgency. Testers own severity; product/business owns priority.
The bug report below is the kind a developer cannot act on. Rewrite it into a proper report with a one-sentence title, numbered steps to reproduce, expected result, actual result, environment, and what evidence you would attach. Invent reasonable specifics for a NZ context.
“Login is broken. Doesn’t work for me. Pls fix.”
Rewrite as a complete bug report:
Show model answer
A complete report (specifics invented but plausible): Title: Login fails with HTTP 500 when the password contains an ampersand (&) on the customer portal. Steps to reproduce: 1. Go to https://portal.example.co.nz/login on Chrome. 2. Enter a valid registered email (e.g. test.user@example.co.nz). 3. Enter a password containing an ampersand, e.g. "Pass&word1". 4. Click "Log in". Expected result: User is authenticated and lands on the account dashboard. Actual result: Page returns an HTTP 500 error and a generic "Something went wrong" message. The user is not logged in. Reproduces every time with an ampersand in the password; logins without special characters succeed. Environment: Chrome 124 on Windows 11; UAT environment, build 2.14.0. Evidence to attach: screenshot of the 500 page, the server log snippet showing the unescaped-character exception, and a screen recording of the reproduction. Why this is better than the original: the original had no steps, no environment, no expected vs actual, and no evidence, so the developer would have to chase the tester for everything. A good report lets the developer reproduce the bug without asking a single question.
A high-severity defect at a NZ payroll provider: holiday-pay calculations are wrong for staff who changed hours mid-year, under-paying them. Walk the defect through the full lifecycle (New → Closed), naming who owns each state and what happens, then run a 5 Whys root cause analysis and name one process improvement that would stop this class of bug recurring.
Show model answer
Lifecycle walkthrough: - New — Tester: logs the defect with steps, expected vs actual, severity High, evidence (a worked example of an under-paid employee). - Assigned — Lead/Manager: triages, confirms severity and priority (compliance + staff under-payment makes it P1), assigns to a developer. - Open / In Progress — Developer: investigates and fixes the mid-year hours-change calculation. - Fixed — Developer: code change made, ready for retest; notes which build contains the fix. - Retest — Tester: confirms the original case now pays correctly, and runs a regression on the surrounding payroll logic. - Closed — Tester: verified fixed in all affected environments; defect closed with a note linking the fix. 5 Whys (example chain): 1. Why was holiday pay wrong? Because the calculation used the latest hours, not the average across the period. 2. Why did it use the latest hours? Because the requirement for mid-year hours changes was never specified. 3. Why was it never specified? Because the Holidays Act averaging rule was not captured in the acceptance criteria. 4. Why was it not captured? Because no one with payroll-compliance knowledge reviewed the story. 5. Why was there no compliance review? Because the team has no checklist step for legislative requirements. Root cause: a missing compliance-review step in the requirements process, not a coding slip. Process improvement: add a mandatory legislative/compliance review (and a Holidays Act test checklist) to the definition of done for any payroll-calculation story, and add boundary test cases for mid-year changes to the regression suite. The point of RCA is to prevent the class of bug, not just patch the one instance.
How this has changed
The field moved. Here is how Defect Management evolved from its origins to current practice.
Bug tracking is literal — engineers keep paper logs or spreadsheets of known defects. No systematic prioritisation. Bug reports are informal memos. The concept of a defect lifecycle (Open → In Progress → Resolved → Closed) does not yet exist as a formal discipline.
Bugzilla created at Netscape (open-sourced 1998). The first widely-used bug tracking system. Defects get IDs, priorities, components, and reproducible steps. Defect management becomes a first-class software engineering practice.
JIRA launched by Atlassian. Combines bug tracking with agile project management. The defect becomes a ticket in a backlog, blurring the line between bugs and features. Teams begin tracking bugs alongside stories in the same workflow.
Defect metrics (escape rate, MTTR, defect density) become standard reporting. Root cause analysis frameworks (5 Whys, fishbone) enter QA practice. The shift from "find and fix" to "understand the systemic cause" begins in mature organisations.
AI tools can triage incoming defect reports, suggest likely root causes from similar historical defects, and propose fixes based on code analysis. The tester's role shifts from defect administration to defect analysis — understanding patterns, systemic causes, and process improvements.
Self-Check
Click each question to reveal the answer.
Why teams fail here
- Bug reports lack reproduction steps, so developers cannot reproduce them and mark them cannot reproduce
- Severity and priority are conflated — a cosmetic bug on the CEO dashboard gets Priority 1
- Defects are closed without re-test evidence, so regressions go undetected
- No root cause analysis means the same class of defect recurs sprint after sprint
Interview Questions
What NZ hiring managers ask about Defect Management — and what strong answers look like.
What makes a bug report actionable, and what are the most common reasons bug reports get ignored or rejected?
Strong answer: An actionable bug report has: a clear title that summarises the problem (not "it's broken"), exact reproduction steps from a known starting state, what actually happened, what was expected, environment details (browser, OS, app version), and supporting evidence (screenshot, log, HAR file). Reports get rejected for: "cannot reproduce" (steps are vague or environment not specified), "by design" (no expected behaviour was stated), "not a bug" (requirement not documented), or "duplicate" (poor search before reporting). I aim for reports a developer can act on without a back-and-forth conversation.
Grad/Junior
How do you decide whether to fix a bug now, defer it, or accept it as a known issue?
Strong answer: The decision depends on four factors: severity (what is the user impact?), likelihood (how many users will hit this?), cost of fix (simple patch or architectural change?), and context (pre-release, production, or experimental feature?). A critical data corruption bug gets fixed before release regardless of effort. A cosmetic glitch in a rarely-used admin panel might be accepted as known. I use a risk-impact matrix: high severity × high likelihood = fix now; low severity × low likelihood = backlog or accept. I document accepted-risk decisions explicitly so they are conscious choices, not oversights.
Junior/Mid
What does a healthy defect removal efficiency (DRE) look like, and how do you use it to improve the testing process?
Strong answer: DRE = defects found before release / (defects found before + after release). A DRE above 95% means fewer than 5% of defects escape to production — generally considered good. I use DRE as a lagging indicator: if it trends downward over several releases, I look for systemic causes — reduced test coverage from time pressure, a component that is under-tested, or a new team member who needs support. I combine DRE with defect escape analysis: where in the SDLC did escaped defects originate? Requirements defects that escape to production suggest earlier review rather than more late-stage testing.
Mid/Senior
Q1: Why is finding a bug not the same as managing a defect?
A defect is only managed once it is captured in a tracker with enough detail to reproduce it, classified by severity and priority, and moved through a defined lifecycle to a verified, closed state. A bug that lives in a chat message is unowned and untracked — it never reaches triage and tends to escape to production.
Q2: What is the single test of a good bug report?
A developer can reproduce the issue without asking you anything. That means a precise title, numbered steps from a known starting state, expected and actual results (with error messages verbatim), environment details, and evidence such as a screenshot, video, or log snippet.
Q3: How do severity and priority differ, and who owns each?
Severity is the technical impact — how badly the system is broken (crash, data loss vs cosmetic). Priority is the business urgency — how soon it must be fixed relative to everything else. They are independent; a low-severity typo can be high priority before a demo. Testers set severity; product/business sets priority.
Q4: What does the “Deferred” state mean, and how is it different from “Rejected”?
Deferred means it is a real defect but won’t be fixed this release — it is consciously scheduled for later. Rejected means it is not a defect at all: by design, a duplicate, or not reproducible. Deferred keeps the bug alive in the backlog; rejected closes it as a non-issue.
Q5: What is the purpose of root cause analysis, and when is it worth doing?
RCA asks “why” not just “what broke” — using techniques like 5 Whys or a fishbone diagram — so you can prevent the whole class of bug recurring, not just patch one instance. It is worth doing for high-severity or recurring defects, and its findings feed into process improvements and updated test checklists.
Q6: Your team is testing a KiwiSaver contribution update on an Benefits NZ portal. A defect is found two days before go-live: switching from voluntary to compulsory contributions sets the rate to 0% instead of 3%. The PM wants to defer it because the fix is risky. How do you handle the defect report and the deferral decision?
A: Log it immediately with Critical severity — a 0% contribution rate causes direct financial harm to members and likely breaches KiwiSaver Act obligations, so this is not a candidate for informal handling. In the tracker, document the financial and compliance impact explicitly (not just "calculation wrong") so the deferral decision carries full context. If the PM still chooses to defer, require written sign-off from the business owner in the defect ticket itself — this protects the team, creates an audit trail, and ensures the risk decision is owned by someone with the authority to make it. Flag the deferred defect in the test summary report as a known high-risk item.
Q7: What is the key difference between defect management and risk-based testing, and how do they work together on a project?
A: Risk-based testing decides where to focus testing effort before and during a test cycle — it is a prioritisation and coverage strategy. Defect management is a tracking and lifecycle discipline that kicks in after a defect is found. They complement each other: the risk assessment shapes which areas get the most test coverage (and therefore where most defects are likely to surface), while defect management ensures every defect found in those areas is captured, classified, and resolved systematically. On a HealthNZ patient data system, for example, risk-based testing might allocate the most coverage to data privacy flows; defect management then ensures any privacy defect found there cannot be quietly dropped or informally "fixed" without a verified close.
Q8: A developer tells you: “I've fixed it on my local environment — you can close the ticket.” What is wrong with this and how do you respond?
A: A fix on a developer's local environment has not been verified by the tester, has not been deployed to the test or staging environment, and has had no regression check run around it. Closing the ticket at this point skips the Retest and regression steps that are the tester's core responsibility. The correct response is to explain that the defect moves to “Fixed” once the developer has deployed the build to the agreed retest environment — then you will retest the original reproduction steps, run a regression sweep on adjacent functionality, and only close the ticket after confirming it is resolved in all affected environments. This is especially critical in NZ payroll or RBNZ-regulated systems where a “fixed” defect that actually escaped to production carries compliance consequences.
Q9: When is it appropriate NOT to log a defect formally in a bug tracker?
A: Formal defect logging adds process overhead, so it is disproportionate in a few specific situations: when you are pair-testing with the developer sitting next to you and they fix a trivial issue in under a minute (capture a session note instead); when testing a throw-away prototype or wireframe where the code will be deleted; or in a solo personal project with no handoff. However, these are narrow exceptions. Any defect in a system with external users, financial or health data, regulatory obligations (Revenue NZ, CoverNZ, RBNZ), or third-party acceptance testing must be formally tracked — informal handling in those contexts is a compliance and audit risk, not just a process preference.
Try It — Severity or priority?
Five bugs have been reported on a NZ banking portal. Assign the correct Severity (Critical/High/Medium/Low) and Priority (P1-Urgent/P2-High/P3-Medium/P4-Low) to each.
| Bug | Severity | Priority |
|---|---|---|
| Payment is processed and money debited, but no confirmation email is sent and the transaction doesn’t appear in transaction history | ||
| Bank logo in the footer is pixelated — CEO demo to the board is tomorrow | ||
| Bulk payment CSV export fails — workaround is to export individual payments one by one | ||
| Session expires after 15 minutes with no warning — RBNZ security guidelines require 20-minute timeout with a warning | ||
| "Help" tooltip on the password field has a typo in a rarely visited screen |
Answers
| Bug | Severity | Priority | Why |
|---|---|---|---|
| Payment debited, no record | Critical | P1-Urgent | Data loss and financial integrity breach. Both are maximum — fix immediately. |
| Pixelated logo, CEO demo tomorrow | Low | P1-Urgent | Classic low-severity / high-priority. A cosmetic bug, but a business deadline makes it urgent today. |
| CSV export fails, workaround exists | Medium | P3-Medium | Function broken but not blocked. Workaround reduces urgency — plan for next sprint. |
| Session timeout non-compliant | High | P1-Urgent | Regulatory compliance failure (RBNZ). Severity is High, but compliance risk makes it P1. |
| Tooltip typo, rarely visited | Low | P4-Low | Cosmetic, low-traffic, no workaround needed. Backlog item. |
The key insight: the logo and compliance bugs both illustrate that severity and priority are independent. Business context drives priority — not just technical impact. Testers own severity; product/business owns priority.
9 Root Cause Analysis
Root cause analysis (RCA) asks not just what broke but why it was possible for it to break. The goal is to surface the systemic cause so the whole class of bug is prevented, not just the one instance patched. Two techniques dominate in software testing: the 5 Whys and the Fishbone (Ishikawa) diagram.
The 5 Whys
Start with the observed symptom and ask “Why?” five times in sequence. Each answer becomes the input to the next question. The chain terminates when you reach a root cause that is a process gap, a missing control, or a design decision — something you can actually change.
When to use it: linear cause chains where one thing leads to the next. Works best when there is a single, identifiable failure path and the team can trace it without needing to map multiple independent factors.
A production incident: users attempting to log in via RealMe on a government portal receive a timeout error. The incident team runs a 5 Whys:
- Why did RealMe logins time out? The authentication service took over 30 seconds to respond and the gateway dropped the connection.
- Why was the authentication service so slow? It was running a full-table scan on the user sessions table on every login attempt.
- Why was it doing a full-table scan? The query filtering on
session_tokenhad no index. - Why was there no index on
session_token? The column was added in a rushed hotfix during the last release and the index creation step was skipped to save time. - Why was the index creation step skipped? The team had no database change checklist in the deployment process — adding an index was discretionary, not mandatory.
Root cause: no mandatory database change checklist in the deployment process. Fix: add a “schema changes verified (indexes, constraints, migrations)” gate to the definition of done for every release.
Fishbone (Ishikawa) Diagram
The fishbone diagram maps multiple contributing factors to a defect across six standard categories. The “head” of the fish is the problem; the “bones” are the categories. You brainstorm causes in each category to identify all the factors that contributed, not just one linear chain.
The six categories, mapped to software:
| Category | In manufacturing | In software testing |
|---|---|---|
| Method | Process or procedure used | Code logic, algorithm, business rule implementation |
| Machine | Equipment used | Infrastructure, database configuration, server settings |
| Material | Raw materials or inputs | Test data, input data quality, seed data, reference data |
| Man | Human factors | Developer misunderstanding of requirements, training gaps, communication failures |
| Measurement | Inspection or testing process | Inadequate test coverage, missing test cases, no regression for this scenario |
| Environment | Physical or operational environment | Environment config drift, deployment differences, timezone/locale settings |
A KiwiSaver provider discovers that contribution calculations are wrong for members who changed their contribution rate mid-month. A fishbone across the six categories finds:
- Method: The calculation used a fixed rate for the entire pay period instead of prorating across the rate-change date — the business logic was wrong.
- Machine: The database stored rates without an effective-date column, so mid-period rate changes overwrote the historical rate rather than appending a new record.
- Material: Test data only covered members who held the same rate for an entire month — no mid-period change scenarios were included in the test dataset.
- Man: The developer who built the calculation had not read the Revenue NZ guidance on proration and assumed the simpler whole-period calculation was correct.
- Measurement: The test plan had no test cases for mid-period rate changes — the scenario was simply not on anyone’s list.
- Environment: The UAT environment used a database seeded with clean annual data and no mid-year transitions — the production scenario never appeared in testing.
Six independent factors all contributed. No single 5 Whys chain would have surfaced all of them. The fishbone forced the team to look at every category and find that the root cause was systemic: a gap in requirements, test data, developer knowledge, test coverage, and environment setup all aligned to let the defect through.
5 Whys vs Fishbone: when to use which
| Technique | Best for | Weakness |
|---|---|---|
| 5 Whys | Single, linear cause chains — one thing led to the next. Fast, low-ceremony, good for incident retrospectives. | Only follows one branch; misses parallel contributing factors. Can stop too early or follow the wrong branch. |
| Fishbone | Multi-factor, systemic issues where several independent causes converged. Good for defects that escaped through multiple gaps. | More time-intensive; requires a facilitated session; can generate too many branches if not kept disciplined. |
10 Defect Taxonomy
Not all defects are alike. Classifying defects by type as well as severity and priority tells you where in the SDLC each class of bug originates — and where to add controls to stop it happening again. When your defect tracker shows 40% of bugs are data defects, that is a signal about data management practices, not just code quality.
| Type | Definition | Examples | NZ example |
|---|---|---|---|
| Functional | The system does the wrong thing or fails to do something it should | Wrong calculation output, missing feature, incorrect business rule | An Revenue NZ tax credit calculator applies the wrong threshold for the independent earner tax credit, under-paying rebates |
| Usability | The system works correctly but is confusing, unclear, or inaccessible | Confusing navigation, poor error messages, ambiguous labels | A KiwiSaver online portal shows “Error 4012” with no explanation when a member enters an invalid Revenue NZ number, leaving users with no idea how to proceed |
| Performance | The system is too slow, consumes excessive resources, or degrades under load | Response time exceeds SLA, memory leak, N+1 database query | An NZ council rates-payment portal times out for 30% of users during end-of-month payment peaks because the rates query runs a full-table scan without pagination |
| Security | A vulnerability that could allow unauthorised access, data exposure, or manipulation | SQL injection, authentication bypass, insecure direct object reference | A health portal allows a logged-in user to view another patient’s appointment history by incrementing the patient ID in the URL — a direct breach of the Health Information Privacy Code |
| Data | Data is stored, processed, or transmitted incorrectly, or referential integrity is broken | Duplicate records, orphaned foreign keys, rounding errors in stored values | A payroll system stores leave balances as FLOAT instead of DECIMAL, causing rounding errors that accumulate to >0.5 days over a year — employees are under-paid leave entitlements in breach of the Holidays Act 2003 |
| Interface | A mismatch between what one system sends and what another system expects | API contract mismatch, wrong field name or type in response, version incompatibility | An CloudBooks payroll integration sends employeeId as a string but the receiving HRIS expects an integer — payslips silently fail to import for any employee with a numeric ID above 999 |
| Environment | The defect is caused by configuration, deployment, or infrastructure rather than application code | Wrong environment variable, missing config file, deployment order error | A production deployment of an Benefits NZ benefit calculation service uses TZ=UTC instead of TZ=Pacific/Auckland — payments scheduled for midnight NZT are processed the previous calendar day, creating incorrect effective dates |
⚠ Common mistake: logging everything as “Functional”
When testers default to classifying every defect as Functional, the taxonomy data is useless. An environment mismatch logged as Functional tells you nothing about the deployment process. A data integrity bug logged as Functional looks like a code problem rather than a schema design issue. Take the thirty seconds to classify correctly — it is the metric that guides process improvement at the team level.
11 Bug Bash
A bug bash is a time-boxed, all-hands exploratory session where everyone on the team — developers, designers, product managers, testers, and sometimes customer-facing staff — stops their normal work and actively tries to break the product for a fixed period, usually two to three hours.
The goal is maximum coverage through diverse perspectives. Developers find edge cases in their own features that no one else would think to try. Designers catch UX flows that break when used by a real person rather than described in a ticket. PMs discover that the feature they specified doesn’t actually do what they meant. Testers surface integration paths that structured test plans miss.
How to run one
- Define scope: tell participants which features, flows, or modules are in scope and which are out. Without scope, people explore the same well-trodden paths they already know.
- Prepare the environment: ensure everyone has test account credentials, access to the environment, and a way to reproduce data states they need. A bug bash that stalls on “I can’t log in” wastes the session.
- Run the session: two to three hours is the practical limit. Energy drops sharply after that. Keep it focused — no normal sprint work during the session.
- Log everything in the tracker: every participant logs what they find directly into Jira (or your tracker of choice) during the session, not in notes afterwards. Defects logged after a bug bash notoriously lose context and steps to reproduce.
- Triage at the end: spend the final 20–30 minutes of the session triaging as a group. Assign severity, mark duplicates, and identify which defects block the release. This is faster as a group when everyone has context from the session.
Bug bashes are common in NZ startups in the week before a major release — the small team size makes it practical for everyone to participate. On government projects (Benefits NZ, HealthNZ, Revenue NZ), bug bashes are sometimes used as informal UAT rounds with policy staff before a formal UAT gate, allowing subject matter experts who know the legislation to try flows that a tester without domain knowledge might not think to run. This is especially valuable for compliance-heavy features: a policy analyst spotting a wrong benefit calculation in a bug bash is far cheaper than finding it in a post-release audit.
When it works
- Near-release, for maximum coverage: the feature set is stable enough that everyone can use it as a real user, not as a broken prototype.
- When developers have not used the feature as a user: engineers often test their own code at the unit level but never click through the full user journey. A bug bash forces that perspective.
- When you have domain experts who are not testers: a payroll specialist, a compliance officer, or a customer-service representative will try things a tester would never think of.
- When you need fast, broad coverage before a hard deadline: a structured test plan for a two-week sprint cannot be completed in two days; a bug bash can give you meaningful coverage in an afternoon.
When it does not work
- If the environment is unstable: if participants spend 40% of the session waiting for environments to recover, the bug bash produces frustration, not defects. Stabilise the environment first.
- If the team has no testing mindset training: participants without any testing background tend to click through the happy path and declare “it works.” A ten-minute briefing on exploratory heuristics (SFDPOT, error guessing) before the session substantially improves defect yield.
- As a substitute for a structured test plan: a bug bash finds things structured testing misses, but it does not replace systematic coverage of requirements. It is a complement, not a replacement.
- On highly technical systems with no UI: an API-only service or a data pipeline has no surface for most non-testers to engage with. Bug bashes work on user-facing surfaces.
⚠ Common mistake: logging bugs in notes “to file later”
Defects captured in personal notes during a bug bash rarely make it into the tracker with enough detail to be actionable. In the noise of a fast-paced session, steps to reproduce are forgotten within minutes. The rule: if you find something, log it in the tracker immediately, even as a rough placeholder with a title and a screenshot. You can refine the report after the session; you cannot recover the context from a sticky note three hours later.