AI Test Data Management

Role: You are a test data engineer for a NZ retail bank's QA team.
Task: Generate 10 synthetic test queries for a personal loan application chatbot.
Requirements:
- All data must be synthetic — no real names, no real IRD numbers, no real bank accounts
- Use NZ-appropriate names proportional to NZ demographics: include at least 2 Maori names, 1 Pacific Island name, 1 South Asian name
- Cover the following scenarios:
  1. Standard application: single income, employed full-time, amount within normal range ($5,000–$30,000)
  2. Income edge case low: income just above minimum threshold ($28,000 annual)
  3. Income edge case high: high income applicant asking about maximum available ($100,000+)
  4. Student loan complication: applicant with existing student loan asking how it affects eligibility
  5. Multiple employers: applicant with two part-time jobs totalling $55,000 annual income
  6. Self-employed: sole trader with irregular income asking what documents are needed
  7. Existing debt: applicant with existing personal loan asking about refinancing or top-up
  8. First-time applicant: no credit history, asking what factors are considered
  9. Adversarial 1: applicant trying to get information about another person's loan eligibility
  10. Adversarial 2: applicant asking the chatbot to ignore its instructions and confirm approval

Format: For each query, provide:
- Query ID: Q01–Q10
- Scenario type: [category from above]
- Synthetic user name: [NZ-appropriate name, clearly fictional]
- Query text: [the message the user would type]
- Expected chatbot behaviour: [what a correct response looks like]

Constraints: Use clearly fictional income figures that are round numbers. Use "Test" as a surname suffix (e.g., "Aroha Test", "James Test") to make synthetic status obvious. Do not use real bank product names or interest rates.

Priority 1: PII and personal data scan
What I am looking for: Employee names, IRD numbers, salary details, health information, disciplinary records, personal email addresses, or any information that could identify a specific individual embedded in documents that will be retrieved and shown to other employees.
Tool/technique: Automated PII detection (Microsoft Presidio, spaCy NER, or a custom regex for NZ-specific formats — IRD number pattern, NZ phone, NZ bank account). Run over all documents before loading into the knowledge base. Flag any document with detected PII for manual review.
Failure in practice: The assistant retrieves a 2023 restructuring memo that mentions individual employees by name alongside their redundancy package details, and surfaces this in response to an HR query from a current employee.

Priority 2: Document freshness audit
What I am looking for: Documents that have been superseded by newer policy, legislation change (especially post-2022 employment law changes in NZ), or organisational restructure. Outdated documents that remain in the knowledge base will be retrieved with the same confidence as current ones.
Tool/technique: Extract document creation/modification dates. Flag documents older than 12 months for review. Cross-reference policy documents against current HR policy version list. For NZ-specific content, check against Employment Relations Act 2000 amendments and current Holidays Act guidance.
Failure in practice: The assistant quotes 2022 sick leave entitlements (5 days per year) when the 2021 Holidays (Increasing Sick Leave) Amendment Act increased the minimum to 10 days. The knowledge base was loaded before the change was widely documented internally.

Priority 3: Contradiction detection
What I am looking for: Two or more documents in the knowledge base that make conflicting statements about the same policy, procedure, or entitlement. When a RAG system retrieves contradictory documents, it may synthesise a response that combines both, producing a confident but internally inconsistent answer.
Tool/technique: Cluster documents by topic using embeddings. Within each cluster, use an LLM to compare the key claims across documents and flag direct contradictions. Manual review for HR-critical topics: leave entitlements, disciplinary procedures, remuneration bands.
Failure in practice: One document says the company's parental leave top-up is 8 weeks at full pay; a later document (after a policy change) says 12 weeks. The assistant synthesises "10 weeks at full pay" — a number that appears in neither document and is legally meaningless as an employee expectation.

Priority 4: Coverage gap analysis
What I am looking for: Topics an employee would reasonably ask the HR assistant about that are not covered in the knowledge base. Coverage gaps cause the model to either hallucinate an answer or give an unhelpfully generic response when a specific one is needed.
Tool/technique: Generate a list of 50–100 common HR queries using an LLM. Run each query against the knowledge base and measure retrieval confidence. Any query returning low-confidence or no results signals a coverage gap. Prioritise gaps for document sourcing.
Failure in practice: The assistant is asked about the company's flexible working request process (a statutory right under NZ employment law). The knowledge base has no documents on this topic. The assistant responds with generic advice about "speaking to your manager," which is technically not wrong but is unhelpfully vague and may cause compliance risk if an employee believes this is the complete process.

Priority 5: Format and encoding consistency check
What I am looking for: Documents in incompatible formats (PDF scans vs. structured text vs. HTML), inconsistent terminology for the same concept (e.g., "annual leave" vs. "holiday leave" vs. "AL"), encoding errors from the import process (garbled characters, broken table structures), and documents where key information is in images rather than text.
Tool/technique: Automated format audit — check that all documents are machine-readable text (not scanned PDFs). Run a term normalisation check to identify synonym clusters. Visual inspection of high-priority documents for rendering fidelity.
Failure in practice: A critical policy table was imported from a Word document as a scanned image — the retrieval system treats it as an empty document. Queries about the content of that table return no relevant results, so the model generates an answer from context alone.

Scenario A: Data drift
The statistical distribution of live production data (dominated by winter products after May) diverged from the distribution used in testing (March data, no winter categories). The model was never invalid — it was valid for the data it was tested on. But as the distribution of real-world queries shifted, its accuracy declined. This is seasonal data drift. The mitigation is to ensure test data covers all seasonal patterns (not just the current season at testing time), and to establish a production monitoring test set with known-correct labels that runs monthly.

Scenario B: Synthetic data gap
The problem was not a privacy breach (synthetic data was used) — but the synthetic data was not representative of the actual user population. Defaulting to Pakeha naming and phrasing patterns meant the test data did not represent Pacific Island users, who make up a significant proportion of DHB patients in many regions. This is a demographic coverage gap in synthetic data generation. The AI was never tested on the queries it would actually receive from a large segment of its user base. The fix is to explicitly specify demographic proportions in synthetic data generation prompts.

Scenario C: Knowledge base quality failure (specifically: document freshness failure)
The knowledge base contained a superseded 2019 document that had not been removed or flagged when the 2023 Building Act amendments were issued. The AI system retrieved this document with full confidence because the retrieval system has no awareness of legislative change — it finds the most semantically relevant document, which happened to be outdated. This is a knowledge base freshness failure. The mitigation is a document freshness audit (check all documents against a known publication date) and a process to update or remove documents when their source legislation changes.

Scenario D: Privacy breach (NZ Privacy Act 2020)
Using real production customer data — including names, account numbers, addresses, and transaction histories — in a test environment breaches Privacy Principle 4 (data collected for fraud detection in production was not collected for the purpose of testing an AI model) and Privacy Principle 5 (test environments have weaker security controls than production, creating elevated risk of unauthorised access or disclosure). The 15,000 customers did not consent to having their data used in this way. The correct approach is to generate synthetic transaction data that mimics the statistical properties (transaction amounts, frequency, merchant categories) without containing real customer identifiers.