Governance & External Audits
Building a testing function that stands up to the most rigorous external scrutiny. Learn by doing.
1 The Hook
A major NZ bank was recently audited by the Reserve Bank. The auditor picked a random transaction bug from six months ago and asked: "Who tested the fix, on what date, in what environment, and where is the screenshot of the database state?"
The team couldn't find it. The result? A formal warning and a six-figure compliance project. In governance, if it isn't documented with evidence, it didn't happen.
2 The Rule
Governance is "Proof of Process." Your goal is to be able to recreate any testing outcome, at any time, for any auditor.
Senior engineer insight
The shift that changed how I think about QA governance: stop treating it as bureaucratic overhead and start treating it as institutional memory. When an OAG auditor asks why a critical payment test was marked Pass in March, your documentation should answer that question without you being in the room — because someday you won't be. The RTM, deviation logs, and sign-off chain are not there to satisfy the auditor; they are there so the next Test Manager can reconstruct exactly what you decided and why.
Most common mistake: maintaining governance artefacts as a "pre-audit sprint" rather than a continuous discipline — by the time the auditor arrives, the gap between what happened and what was recorded is too wide to close honestly.
From the field
On a TMMi Level 3 assessment for a NZ central government agency, the assessor asked to see evidence that testing was independent — not just asserted in the Test Strategy, but demonstrable from the defect records. The team had documented independence in the policy but testers and developers were sharing Jira accounts, so every "Verified" status was attributed to the same login that raised the code change. The assessors flagged it immediately. Within the week we introduced role-separated accounts, a formal re-test workflow, and an "Independence Attestation" field on the sign-off template. The lesson that generalises: governance controls need to be visible in the artefacts themselves, not just stated in policy — an auditor cannot take your word for it, they need to see it in the data trail.
3 Watch Me Do It: The "Audit Trail" Verification
Observe how a Test Manager verifies the "Provenance" of a test result before a major release. This is the checklist for an audit-proof sign-off.
-
Requirement Trace: Does Requirement #101 link to Test Case #505?
EVIDENCE: RTM (Requirements Traceability Matrix) -
Environment Check: Was the test run in 'SIT-02' (Prod-like) or just on a dev's laptop?
EVIDENCE: Environment Snapshot Log -
Result Integrity: Is there a screenshot of the "Success" message AND a log of the database entry?
EVIDENCE: Execution Artifacts -
Sign-off Accountability: Who approved the final state? Was it the Business Owner or just the Dev?
EVIDENCE: Approval Timestamp / Sign-off Email
4 Compliance Lab: The Audit Challenge
In this lab, you must audit a "Legacy" project that is about to launch. You have limited time to find the gaps before the external auditor arrives.
Your Task: The Pre-Audit Check
You find 50 "Passed" tests. However, 10 of them have no screenshots, and 5 were run in a "Mock" environment that doesn't match Production.
WORK THROUGH THESE STEPS:
- Categorise the Risk: Which missing evidence is "Critical" (e.g. Payments) vs. "Low" (e.g. UI).
- Action Plan: Do you re-run the tests? Or do you document a "Deviation" and risk-accept it?
- The Privacy Test: You notice a tester used their own personal credit card for a test. What is the legal implication in NZ?
Draft a "Deviation Statement." Explain why the Mock environment was used and how the risk is mitigated. Write it in your notes.
5 ISO 25010 & NZISM Compliance Mapping
Auditors use international standards to judge your testing rigor. Knowing these frameworks helps you structure evidence that will pass inspection.
ISO 25010: Software Quality Dimensions
ISO 25010 defines 8 characteristics of quality software. Map your tests to these to show you're covering all dimensions.
| Quality Dimension | What It Means | Test Examples |
|---|---|---|
| Functional Suitability | Does it do what it's supposed to do? | Happy-path tests, requirement traceability |
| Reliability | Does it crash? Does it corrupt data? | Stress testing, data integrity checks, failover tests |
| Operability | Can ops teams run it? Is it maintainable? | Health checks, monitoring, runbook verification |
| Security | Can attackers break in? Is data encrypted? | OWASP Top 10 scanning, penetration testing, compliance checks |
| Compatibility | Works on different OSs, browsers, devices? | Cross-browser testing, API compatibility, backward compatibility |
| Performance Efficiency | Is it fast enough under load? | Load testing, response time thresholds, resource usage |
| Maintainability | Can developers fix bugs? Is code documented? | Code review, documentation audit, deployment verification |
| Transferability | Can the system be migrated to new hardware, cloud, etc? | Migration testing, data export/import verification |
NZISM Alignment Checklist (NZ Government)
If your client is a government agency or state sector, they must comply with NZISM (NZ Information Security Manual).
Key NZISM Controls for Test Managers:
- BC-2: Business Continuity Plan exists and is tested annually with evidence kept.
- AC-1: Access control is tested. Only authorized users can access systems. Test user provisioning and de-provisioning.
- AC-2: Audit logging is tested. System logs user activity. Verify logs are tamper-proof.
- CR-1: Encryption in transit and at rest. Test TLS, certificate validation, encrypted database storage.
- IM-2: Data Integrity controls. Test database constraints, checksums, audit trails.
- SY-1: Secure development baseline. Code is scanned for vulnerabilities before release.
6 Internal Audit Checklist & Documentation Template
Before external auditors arrive, run your own audit using this checklist. Fix gaps while you can.
Pre-Audit Self-Check (20 items)
| Item | Status | Evidence Location |
|---|---|---|
| Requirements Traceability Matrix (RTM) complete | [ ] Yes [ ] No | __________ |
| All test cases are documented (not just automated) | [ ] Yes [ ] No | __________ |
| Test executions logged with date, tester, environment, result | [ ] Yes [ ] No | __________ |
| Screenshots or log outputs saved for all "Pass" results | [ ] Yes [ ] No | __________ |
| All bugs logged in defect tracking system with reproduce steps | [ ] Yes [ ] No | __________ |
| Bug fixes verified and re-tested in same environment | [ ] Yes [ ] No | __________ |
| Test data generation documented (synthetic vs. masked production) | [ ] Yes [ ] No | __________ |
| Environment configurations documented (OS, DB version, patches) | [ ] Yes [ ] No | __________ |
| Risk assessment completed for deviations from test plan | [ ] Yes [ ] No | __________ |
| Sign-off obtained from business owner or product manager | [ ] Yes [ ] No | __________ |
| Test metrics tracked (coverage %, bugs found, severity distribution) | [ ] Yes [ ] No | __________ |
| Security testing performed (OWASP Top 10 or equivalent) | [ ] Yes [ ] No | __________ |
| Performance testing done if applicable (load, stress, endurance) | [ ] Yes [ ] No | __________ |
| Compatibility testing across supported browsers/devices | [ ] Yes [ ] No | __________ |
| Regression test suite maintained and run before release | [ ] Yes [ ] No | __________ |
| Test independence verified (testers ≠ developers) | [ ] Yes [ ] No | __________ |
| Known issues documented with risk acceptance sign-off | [ ] Yes [ ] No | __________ |
| Test team training records maintained | [ ] Yes [ ] No | __________ |
| Change log shows all post-release updates and re-testing | [ ] Yes [ ] No | __________ |
| Lessons learned captured for next project | [ ] Yes [ ] No | __________ |
Compliance Documentation Template
Create one of these for each release.
TEST EXECUTION SIGN-OFF DOCUMENT
Release: v2.1.0 | Date: 2026-04-24 | Tester: Sarah Chen | QA Manager: James Park
Test Coverage: 95 test cases executed. 3 bugs found (2 critical, 1 medium). All critical bugs fixed and verified.
Environments Tested: SIT-02 (Linux, PostgreSQL 13), UAT-01 (Windows, Oracle 19c)
Risk Assessment: 1 known issue remains (payment confirmation email delay on international transactions). Severity: Low. Business accepted risk. See Deviation Log v3.
Sign-Off: I certify this release has been tested according to the Test Plan. Based on testing results, the system is ready for production deployment.
Tester: Sarah Chen _________________ Date: 2026-04-24
QA Manager: James Park _________________ Date: 2026-04-24
7 NZ Government Procurement Assessment
If you're selling to NZ government or state sector (health, education, councils), they evaluate quality using GETS (Government Electronic Tenders Service) criteria.
GETS Evaluation Criteria for Quality
When submitting a tender response, prepare evidence for these points:
| Criterion | What Government Wants | Your Evidence |
|---|---|---|
| Testing Maturity | Proof you follow ISTQB or similar methodology | Show training certs, test strategy doc, RTM |
| Quality Metrics | Historical data: defect escape rate, test coverage % | Show past project test summaries, metrics dashboard |
| Security Testing | Proof you test for OWASP vulnerabilities | Show security test results, penetration test reports |
| Change Control | You have a process for managing post-release changes | Show change log, re-test checklist, sign-off process |
| Compliance Alignment | You align with NZISM, Privacy Act 2020, Govt Standards | Show compliance matrix, audit findings, remediation |
Pro Tip: Include a "Quality Commitment" statement in your tender response. Example: "We guarantee 95%+ test coverage for critical paths. If we fall below this, we will extend testing at no cost until we meet the threshold."
8 Common Mistakes
⚠ "Reconstructing" Evidence after the fact
Why it fails: If an auditor sees screenshots with "current" dates for a test run 6 months ago, you have lost all credibility. Never fake evidence. Document the gap honestly.
⚠ Using Real PII in Test
Why it fails: Violates the NZ Privacy Act 2020. You could face legal action. Always use synthetic data or masked production data.
⚠ Assuming Compliance is IT's Job
Why it fails: Quality governance is shared. Your tests prove compliance. If you don't document it, IT can't prove it, and the auditor finds the gap.
Why teams fail here
- Evidence is captured per-project with no consistent naming or storage convention, so when an OAG auditor asks for test results from 18 months ago the team spends days hunting across SharePoint, email threads, and personal drives — and still can't find half of it.
- Deviation logs are created after the fact, often on the eve of the audit, which destroys their credibility — auditors check metadata and creation timestamps, and a deviation log dated the day before their arrival is worse than no log at all.
- TMMi and NZISM compliance are treated as one-off certification exercises rather than embedded practices, so process maturity regresses between assessments and the next audit finds the same gaps as the last one.
- Test Managers sign off releases without confirming that business sign-off was obtained independently — so when a defect escapes to production, there is no audit trail showing who accepted the residual risk, and accountability defaults to QA.
Key takeaway
Governance is not something you do for auditors — it is the proof that your testing function made deliberate, traceable decisions under pressure, and that distinction is exactly what separates a professional QA practice from a team that just clicks buttons.
Enterprise reality
OAG audits, internal assurance reviews, TMMi assessments
- Audit readiness is a continuous state, not a pre-audit sprint — OAG field teams arrive with a 48-hour notice window, and evidence packs assembled from scratch in that window are visibly incomplete and lack the metadata timestamps that prove contemporaneous capture.
- Internal assurance reviews require a single authoritative source of truth: when programme governance, enterprise risk, and QA all maintain separate artefact stores, auditors find version conflicts between them and flag the discrepancy as a control weakness — not the gap itself, but the fact that no one noticed.
- TMMi Level 3 and above demands that your QA governance framework is formally integrated into programme governance — test entry/exit criteria, release gates, and deviation approvals must appear in the programme RAID log and steering committee minutes, not only in QA-owned documents.
- Audit findings at organisational scale become improvement backlog items with named owners, due dates, and escalation paths — without that structure, the same finding recurs at the next assessment cycle, which is the fastest way to lose assessor confidence and trigger unannounced follow-up reviews.
9 Self-Check
Q1. What is a "Deviation Log"?
A document that records every time the team had to move away from the "Standard Process" and why. It proves the team was acting intentionally, not negligently.
Q2. Why is "Independence" a core part of governance?
Because you cannot "mark your own homework." The person testing should not be the person who wrote the code, to ensure an unbiased result.
Q3. What does ISO 25010 cover?
Eight dimensions of software quality: Functionality, Reliability, Operability, Security, Compatibility, Performance, Maintainability, Transferability. Map your tests to these dimensions to show holistic coverage.
Q4. If you're bidding for a government contract, what proof should you provide?
NZISM alignment, test metrics (coverage %, defect escape rate), security test results, and a quality commitment statement showing your testing standards.