Governance & External Audits
Building a testing function that stands up to the most rigorous external scrutiny. Learn by doing.
1 The Hook
A major NZ bank was recently audited by the Reserve Bank. The auditor picked a random transaction bug from six months ago and asked: "Who tested the fix, on what date, in what environment, and where is the screenshot of the database state?"
The team couldn't find it. The result? A formal warning and a six-figure compliance project. In governance, if it isn't documented with evidence, it didn't happen.
2 The Rule
Governance is "Proof of Process." Your goal is to be able to recreate any testing outcome, at any time, for any auditor.
3 Watch Me Do It: The "Audit Trail" Verification
Observe how a Test Manager verifies the "Provenance" of a test result before a major release. This is the checklist for an audit-proof sign-off.
-
Requirement Trace: Does Requirement #101 link to Test Case #505?
EVIDENCE: RTM (Requirements Traceability Matrix) -
Environment Check: Was the test run in 'SIT-02' (Prod-like) or just on a dev's laptop?
EVIDENCE: Environment Snapshot Log -
Result Integrity: Is there a screenshot of the "Success" message AND a log of the database entry?
EVIDENCE: Execution Artifacts -
Sign-off Accountability: Who approved the final state? Was it the Business Owner or just the Dev?
EVIDENCE: Approval Timestamp / Sign-off Email
4 Compliance Lab: The Audit Challenge
In this lab, you must audit a "Legacy" project that is about to launch. You have limited time to find the gaps before the external auditor arrives.
Your Task: The Pre-Audit Check
You find 50 "Passed" tests. However, 10 of them have no screenshots, and 5 were run in a "Mock" environment that doesn't match Production.
WORK THROUGH THESE STEPS:
- Categorise the Risk: Which missing evidence is "Critical" (e.g. Payments) vs. "Low" (e.g. UI).
- Action Plan: Do you re-run the tests? Or do you document a "Deviation" and risk-accept it?
- The Privacy Test: You notice a tester used their own personal credit card for a test. What is the legal implication in NZ?
Draft a "Deviation Statement." Explain why the Mock environment was used and how the risk is mitigated. Write it in your notes.
5 ISO 25010 & NZISM Compliance Mapping
Auditors use international standards to judge your testing rigor. Knowing these frameworks helps you structure evidence that will pass inspection.
ISO 25010: Software Quality Dimensions
ISO 25010 defines 8 characteristics of quality software. Map your tests to these to show you're covering all dimensions.
| Quality Dimension | What It Means | Test Examples |
|---|---|---|
| Functional Suitability | Does it do what it's supposed to do? | Happy-path tests, requirement traceability |
| Reliability | Does it crash? Does it corrupt data? | Stress testing, data integrity checks, failover tests |
| Operability | Can ops teams run it? Is it maintainable? | Health checks, monitoring, runbook verification |
| Security | Can attackers break in? Is data encrypted? | OWASP Top 10 scanning, penetration testing, compliance checks |
| Compatibility | Works on different OSs, browsers, devices? | Cross-browser testing, API compatibility, backward compatibility |
| Performance Efficiency | Is it fast enough under load? | Load testing, response time thresholds, resource usage |
| Maintainability | Can developers fix bugs? Is code documented? | Code review, documentation audit, deployment verification |
| Transferability | Can the system be migrated to new hardware, cloud, etc? | Migration testing, data export/import verification |
NZISM Alignment Checklist (NZ Government)
If your client is a government agency or state sector, they must comply with NZISM (NZ Information Security Manual).
Key NZISM Controls for Test Managers:
- BC-2: Business Continuity Plan exists and is tested annually with evidence kept.
- AC-1: Access control is tested. Only authorized users can access systems. Test user provisioning and de-provisioning.
- AC-2: Audit logging is tested. System logs user activity. Verify logs are tamper-proof.
- CR-1: Encryption in transit and at rest. Test TLS, certificate validation, encrypted database storage.
- IM-2: Data Integrity controls. Test database constraints, checksums, audit trails.
- SY-1: Secure development baseline. Code is scanned for vulnerabilities before release.
6 Internal Audit Checklist & Documentation Template
Before external auditors arrive, run your own audit using this checklist. Fix gaps while you can.
Pre-Audit Self-Check (20 items)
| Item | Status | Evidence Location |
|---|---|---|
| Requirements Traceability Matrix (RTM) complete | [ ] Yes [ ] No | __________ |
| All test cases are documented (not just automated) | [ ] Yes [ ] No | __________ |
| Test executions logged with date, tester, environment, result | [ ] Yes [ ] No | __________ |
| Screenshots or log outputs saved for all "Pass" results | [ ] Yes [ ] No | __________ |
| All bugs logged in defect tracking system with reproduce steps | [ ] Yes [ ] No | __________ |
| Bug fixes verified and re-tested in same environment | [ ] Yes [ ] No | __________ |
| Test data generation documented (synthetic vs. masked production) | [ ] Yes [ ] No | __________ |
| Environment configurations documented (OS, DB version, patches) | [ ] Yes [ ] No | __________ |
| Risk assessment completed for deviations from test plan | [ ] Yes [ ] No | __________ |
| Sign-off obtained from business owner or product manager | [ ] Yes [ ] No | __________ |
| Test metrics tracked (coverage %, bugs found, severity distribution) | [ ] Yes [ ] No | __________ |
| Security testing performed (OWASP Top 10 or equivalent) | [ ] Yes [ ] No | __________ |
| Performance testing done if applicable (load, stress, endurance) | [ ] Yes [ ] No | __________ |
| Compatibility testing across supported browsers/devices | [ ] Yes [ ] No | __________ |
| Regression test suite maintained and run before release | [ ] Yes [ ] No | __________ |
| Test independence verified (testers ≠ developers) | [ ] Yes [ ] No | __________ |
| Known issues documented with risk acceptance sign-off | [ ] Yes [ ] No | __________ |
| Test team training records maintained | [ ] Yes [ ] No | __________ |
| Change log shows all post-release updates and re-testing | [ ] Yes [ ] No | __________ |
| Lessons learned captured for next project | [ ] Yes [ ] No | __________ |
Compliance Documentation Template
Create one of these for each release.
TEST EXECUTION SIGN-OFF DOCUMENT
Release: v2.1.0 | Date: 2026-04-24 | Tester: Sarah Chen | QA Manager: James Park
Test Coverage: 95 test cases executed. 3 bugs found (2 critical, 1 medium). All critical bugs fixed and verified.
Environments Tested: SIT-02 (Linux, PostgreSQL 13), UAT-01 (Windows, Oracle 19c)
Risk Assessment: 1 known issue remains (payment confirmation email delay on international transactions). Severity: Low. Business accepted risk. See Deviation Log v3.
Sign-Off: I certify this release has been tested according to the Test Plan. Based on testing results, the system is ready for production deployment.
Tester: Sarah Chen _________________ Date: 2026-04-24
QA Manager: James Park _________________ Date: 2026-04-24
7 NZ Government Procurement Assessment
If you're selling to NZ government or state sector (health, education, councils), they evaluate quality using GETS (Government Electronic Tenders Service) criteria.
GETS Evaluation Criteria for Quality
When submitting a tender response, prepare evidence for these points:
| Criterion | What Government Wants | Your Evidence |
|---|---|---|
| Testing Maturity | Proof you follow ISTQB or similar methodology | Show training certs, test strategy doc, RTM |
| Quality Metrics | Historical data: defect escape rate, test coverage % | Show past project test summaries, metrics dashboard |
| Security Testing | Proof you test for OWASP vulnerabilities | Show security test results, penetration test reports |
| Change Control | You have a process for managing post-release changes | Show change log, re-test checklist, sign-off process |
| Compliance Alignment | You align with NZISM, Privacy Act 2020, Govt Standards | Show compliance matrix, audit findings, remediation |
Pro Tip: Include a "Quality Commitment" statement in your tender response. Example: "We guarantee 95%+ test coverage for critical paths. If we fall below this, we will extend testing at no cost until we meet the threshold."
8 Common Mistakes
⚠ "Reconstructing" Evidence after the fact
Why it fails: If an auditor sees screenshots with "current" dates for a test run 6 months ago, you have lost all credibility. Never fake evidence. Document the gap honestly.
⚠ Using Real PII in Test
Why it fails: Violates the NZ Privacy Act 2020. You could face legal action. Always use synthetic data or masked production data.
⚠ Assuming Compliance is IT's Job
Why it fails: Quality governance is shared. Your tests prove compliance. If you don't document it, IT can't prove it, and the auditor finds the gap.
9 Self-Check
Q1. What is a "Deviation Log"?
A document that records every time the team had to move away from the "Standard Process" and why. It proves the team was acting intentionally, not negligently.
Q2. Why is "Independence" a core part of governance?
Because you cannot "mark your own homework." The person testing should not be the person who wrote the code, to ensure an unbiased result.
Q3. What does ISO 25010 cover?
Eight dimensions of software quality: Functionality, Reliability, Operability, Security, Compatibility, Performance, Maintainability, Transferability. Map your tests to these dimensions to show holistic coverage.
Q4. If you're bidding for a government contract, what proof should you provide?
NZISM alignment, test metrics (coverage %, defect escape rate), security test results, and a quality commitment statement showing your testing standards.