Managing AI Risks in Testing

1 The Hook

An NZ telecommunications company asked their QA team to use AI to generate test cases for a new billing system migration. The team used AI effectively — 400 test cases in a day. When they ran them, 340 passed. The project manager celebrated: 85% pass rate.

But three months after go-live, customers started receiving double charges. The investigation revealed that 18 of the 60 "failed" test cases had been dismissed as AI hallucinations — invented test conditions that seemed unrealistic. Twelve of those 18 were actually valid edge cases the team had not considered. The AI was right. The team had dismissed real test cases because they assumed AI output was unreliable.

This is not an argument against using AI. The team was right to use AI — 400 test cases in a day is a genuine productivity gain. The failure was not the tool. The failure was a lack of understanding about how AI tools behave and how to review their output critically.

AI tools in testing introduce a specific and learnable set of risks: hallucination (confident but incorrect output), reasoning errors (right facts, wrong conclusions), bias in what the model generates, data privacy hazards when test inputs contain real customer data, and non-determinism that makes test artefacts hard to reproduce.

Testers who understand these risks do not avoid AI — they work with it more effectively. They know which outputs to trust, which to verify, and which to question. That is the skill this module builds.

2 The Rule

3 The Analogy

4 Hallucinations and Reasoning Errors

CT-GenAI-3.1.1, 3.1.2, 3.1.3, 3.1.4

What is a hallucination?

An LLM hallucination is when the model generates factually incorrect content with apparent confidence. The model does not "know" it is wrong — it produces the most statistically likely continuation of its input, regardless of whether that continuation is true. In a testing context, this means:

• Field names that do not exist in your system
• Status codes that are wrong (e.g., the model says the endpoint returns 201, but it returns 200)
• Business rules that are plausible but incorrect (e.g., "IRD numbers are always 9 digits" — they are actually 8 or 9)
• Test data that looks valid but fails your system's validation (e.g., an NZ mobile number formatted as a landline)

Reasoning errors (CT-GenAI-3.1.1)

Reasoning errors are distinct from hallucinations. The model may have the correct facts but draw the wrong conclusion. A common example: given a field with valid range 0–100, a reasoning error generates boundary tests at -1 and 101 but omits tests at 0 and 100 — the actual boundary values. The model understood boundary value analysis in principle but misapplied it.

Bias in AI output (CT-GenAI-3.1.1)

• Training data bias: if the training data was dominated by US software examples, the model may default to US formats — ZIP codes, SSNs, MM/DD/YYYY dates — even when you explicitly ask for NZ context
• Confirmation bias in output: AI tends to generate positive test cases (happy path) over negative ones unless explicitly instructed otherwise
• Recency bias: the model may be more familiar with older API patterns or deprecated libraries than current standards

Identifying hallucinations (CT-GenAI-3.1.2)

Mitigation strategies (CT-GenAI-3.1.3)

• Always provide the actual spec as context — models hallucinate significantly less when given ground truth to work from
• Use few-shot prompting with verified examples — provide one or two correct test cases to establish the right pattern before asking the model to generate more
• Ask the model to cite its reasoning: "For each test case, explain which requirement it is testing" — this surfaces hallucinations quickly because the model cannot cite a requirement that does not exist
• Cross-reference generated test cases against the actual spec before adding them to your test suite

Non-determinism (CT-GenAI-3.1.4)

Running the exact same prompt twice will produce different outputs. This is an inherent property of how LLMs sample from probability distributions. It makes regression testing of AI-generated artefacts challenging — you cannot "re-run" an AI generation session and expect identical results.

• Set temperature to 0 (or the lowest available value) for the most deterministic output possible
• Use seed values if the API supports them
• Save and version the output of AI generation sessions — the saved output is your test artefact, not the prompt

5 Data Privacy and Security

CT-GenAI-3.2.1, 3.2.2, 3.2.3

NZ Privacy Act 2020 context

Under the NZ Privacy Act 2020, personal information must only be collected for a specific, lawful purpose — and individuals have not consented to their data being sent to an AI provider for model training or inference logging. Pasting real customer data into a public AI tool likely breaches Privacy Principle 4 (limiting collection to authorised purposes) and possibly Principle 5 (storage and security of personal information).

What counts as sensitive data in a testing context

• Customer names, email addresses, phone numbers, dates of birth
• IRD numbers (New Zealand's unique national tax identifier — highly sensitive)
• Bank account numbers and payment card details
• Health information of any kind
• Government-issued ID details (passport number, driver licence number)
• Proprietary system specifications (commercially sensitive even if not personal data)

Data privacy risks when using AI for testing (CT-GenAI-3.2.1, 3.2.2)

Mitigation strategies (CT-GenAI-3.2.3)

• Use only AI tools approved by your organisation's security and privacy team
• Replace real customer data with synthetic equivalents — fake NZ names, fake IRD numbers, fake bank account numbers — before prompting
• Use on-premise or private-deployment LLM solutions for sensitive government and health contexts
• Review the AI provider's data processing agreement (DPA) before first use
• Never commit API keys to version control — use environment variables or a secrets manager

6 Environmental Impact

CT-GenAI-3.3.1

LLM inference has a measurable carbon footprint. A single complex prompt to a large general-purpose model may consume as much energy as running several automated tests. For individual use this is negligible, but at scale — generating test cases for thousands of user stories across a large organisation — the environmental cost accumulates.

Practical implications: prefer smaller, purpose-specific models for routine generation tasks over large general models; batch prompts where possible rather than submitting them one by one; cache useful outputs rather than regenerating identical prompts; and factor this into your organisation's AI strategy when selecting tooling (see CT-GenAI Ch 5).

7 AI Regulations and Standards

CT-GenAI-3.4.1

The regulatory landscape for AI is evolving rapidly. NZ testers need to be aware of the frameworks that apply to their context:

For testers: understand which standards apply in your organisation. Ensure AI-assisted test artefacts are documented and traceable — regulators may ask how test coverage was achieved, and "the AI generated it" is not a sufficient audit trail without human review records.

8 Common Mistakes

9 Now You Try

Three graded exercises. Each targets a different AI risk from this chapter. Use the AI to check your answers, then compare to the model answer.

TC-03: NZISM minimum password length is 16 characters (not 8). The NZ Information Security Manual (NZISM) v3.7 specifies a minimum of 16 characters for government systems. "8 characters" is a commonly hallucinated value that reflects older international standards, not NZISM.

TC-06: The test case says "last 5 passwords" in the expected result but "last 3 passwords" in the description. More importantly, the specific history depth (3 or 5) is an invented business rule — neither value is a universal standard and neither is specified in the scenario. The AI invented a specific number with false confidence.

TC-07: ANZ (and most NZ retail banks) do not have a published maintenance window that blocks password reset at 2–4am NZST. This is a plausible-sounding operational rule that does not exist. The AI generated it because "maintenance window" is a common pattern in banking systems — but applied it without any source in the spec.

Role: You are a senior tester for an NZ tax authority digital services team.
Context: The system is an online income tax assessment portal. The income assessment form calculates PAYE deductions and student loan repayments based on annual employment income.
Instruction: Generate boundary value and equivalence partition test cases for the income assessment form.
Input data (synthetic — no real personal data):
- Test user: [synthetic name], IRD number format: 9 digits (e.g. 049-000-001 — a known test value, not a real IRD number)
- Income range: $0 to $200,000 annual employment income
- Student loan repayment threshold: $22,828 (2025–26 threshold per IRD)
- PAYE rates: apply standard NZ tax rates for the income bands tested
- Student loan deduction rate: 12 cents per dollar over the threshold
Constraints: Use only synthetic test data. Do not use real names, real IRD numbers, real addresses, or real employer names. All values must be either publicly documented thresholds or clearly fictional.
Output format: Table with columns: Test ID | Scenario | Input values | Expected PAYE | Expected student loan deduction | Pass/Fail criteria

Privacy violations in original prompt:
1. Real name (John Smith) — use "Test User A" or a generic label
2. Real IRD number format (049-123-456) — use known test/synthetic values
3. Real date of birth — not needed for this test; remove entirely
4. Real address (14 Aroha Street Petone) — use "Test Address NZ" or omit
5. Real employer (Fletcher Building) — use "Employer A" or omit
6. Specific real salary ($87,500) — acceptable to use as a test value but should be labelled synthetic
7. Student loan balance ($23,400) — specific individual balance; use threshold-relative values instead

Problem A: Shadow AI + Privacy breach (NZ Privacy Act 2020)
Using the free tier of ChatGPT without organisational approval is Shadow AI — an unapproved tool handling work data. Pasting real Toka Tū Ake EQC claim numbers, property addresses, and photos of real claimants into a public AI tool is a Privacy Act 2020 breach: the claimants have not consented to their personal information being sent to a third-party AI provider. This likely breaches Privacy Principles 5 (security safeguards) and 11 (disclosure to overseas entities).

Problem B: Hallucination
The AI generated a specific dollar figure ($300,000) that does not match current Toka Tū Ake EQC legislation. The correct cap is $150,000 for land and $150,000 for buildings (plus GST each) under the Earthquake Commission Act 2022. The AI's confident but incorrect figure is a classic hallucination — plausible, specific, and wrong. Using this test case would mean the test suite validates incorrect business logic.

Problem C: Non-determinism
The same prompt producing three different outputs on three runs is a direct consequence of LLM non-determinism (temperature > 0). The mitigation is to save and version the first acceptable output rather than re-running. The test lead should pick one output, review it against the spec, and commit it to the regression suite — not run the prompt again hoping for a "better" version.

Problem D: Non-determinism (with a misguided workaround)
This is also non-determinism, but with a dangerous response: the tester is averaging outputs. You cannot average test cases — each run produces a distinct set of scenarios. Averaging them creates a hybrid set that was never validated as coherent. The correct response is: set temperature to 0 if the tool allows it, or accept one run's output after review. The time-of-day variation is likely server-side randomness, not a meaningful signal.

10 Self-Check

Click each question to reveal the answer.

11 Interview Prep

Real questions asked in NZ QA interviews. Read the model answers, then practise your own version.