AI in Testing
AI won't replace testers. But testers who use AI will replace testers who don't. Learn what's real, what's hype, and how to pilot responsibly.
1 The Hook — Why This Matters
In 2025, a NZ e-commerce company invested $85K in an AI-powered test generation tool. The vendor promised "80% reduction in test creation time." The reality: the generated tests were syntactically correct but asserted incorrect behaviours. One AI-generated test validated that a "Sold Out" badge appeared on in-stock products. It passed for three weeks before a customer reported the bug. The "time savings" were consumed by manual review and correction of AI output.
AI in testing is powerful but immature. The architects who succeed are those who pilot carefully, measure honestly, and never delegate accountability to a model.
2 The Rule — The One-Sentence Version
Use AI to augment human judgment, not replace it. Pilot one capability on one product line before enterprise rollout. Measure real maintenance reduction, not vendor claims.
AI-generated tests can hallucinate incorrect assertions. Self-healing locators fix symptoms, not root causes. Visual AI flags changes but can't judge whether they're bugs. The human remains accountable for quality.
3 The Analogy — Think Of It Like...
An autopilot in a plane.
Autopilot handles 95% of flight beautifully. But the pilot must monitor it, intervene when conditions change, and take full control when the unexpected happens. AI in testing is the same: it reduces routine workload, but the tester must verify its output, catch its errors, and own the final quality decision. No airline removes the pilot because autopilot exists.
Senior engineer insight
The moment that changed how I think about AI in testing: we deployed Mabl across a TransitNZ client portal and measured a genuine 38% drop in locator-related CI failures — but discovered three months in that a critical business rule regression had been silently passing because the AI had "healed" onto a visually-similar but semantically-wrong element. The audit trail was opaque, and nobody had reviewed the healed locators. From that point on, every AI-assisted tool gets a weekly human audit, not just a green dashboard.
Most common mistake: treating a green AI test run as equivalent to a green human-reviewed test run — they are not the same signal.
From the field
A NZ health sector client (primary care network, ~60 clinics) was sold an AI test generation platform on the promise it would halve their test authoring time ahead of a patient portal migration. The team assumed the generated tests were "good enough to run" with light review. What they discovered: the AI had learned from existing test fixtures that pre-populated specific patient IDs, so every generated test was structurally valid but only passed against seeded data — in production data ranges, they failed immediately. The migration release was delayed three weeks while the team rewrote 40% of the AI-generated suite from scratch. The lesson that generalises: AI learns the shape of your test data, not the shape of your business rules. If your training environment is synthetic, your generated tests will be too.
4 Watch Me Do It — Step by Step
Here is how to evaluate and adopt AI capabilities responsibly.
- Self-healing locators
ML models analyse DOM structure and element attributes. When a locator fails, the engine searches for the "closest match" based on visual position, text content, and hierarchy.
Tool Approach Maturity Healenium (EPAM) ML-based locator recovery Mature, widely adopted Testim (Tricentis) Smart Locators with AI grouping Enterprise-grade Mabl Auto-healing + intelligent waits Strong for SaaS/web Realistic expectation for NZ teams: 30-50% reduction in locator-related failures. Vendor claims of 60-80% are optimistic for high-velocity UIs.
- Visual AI (Applitools / Chromatic)
Captures baseline screenshots and uses AI to compare new screenshots. Ignores anti-aliasing and rendering differences; flags meaningful layout changes.
Best for: design system regression, responsive layout verification, CMS-driven content changes, localisation layout breaks.
- Risk-based test selection (Launchable, Appsurify)
ML models predict which tests are most likely to fail based on code change diff, historical failure correlation, and code complexity. Run ML-selected tests on every PR; full suite nightly.
Caution: Risk-based selection is a complement, not replacement, for full regression. Never skip the full suite before release.
- AI-assisted test generation
Capability 2026 State Gherkin to executable scripts Usable with review Tests from Figma/screenshots Pilot phase Session recording to scripts Mature for simple flows OpenAPI to API tests Production-ready
5 When to Use It / When NOT to Use It
✅ Pilot AI when...
- Locator maintenance consumes >30% of team time
- Visual regression is a known pain point
- Test suite is too large for full PR execution
- Budget allows for 3-month pilot with exit option
❌ Avoid AI when...
- Accountability for failures is unclear
- Team lacks bandwidth to review AI output
- Vendor claims seem unrealistic
- Budget doesn't cover pilot + potential failure
6 Common Mistakes — Don't Do This
🚫 Believing vendor claims uncritically
I used to thought: "80% reduction in test creation time" means 80% less work.
Actually: AI-generated tests require human review, correction, and ongoing maintenance. The real savings are typically 20-40% for mature use cases, not 80%. Always measure your own numbers.
🚫 Treating self-healing as a root cause fix
I used to think: Self-healing means I don't need to maintain locators.
Actually: Self-healing fixes symptoms (broken locators) but not root causes (unstable UI, missing test IDs, frequent redesigns). If the UI is genuinely broken, a self-healing test might pass by matching the wrong element. Always investigate why locators break.
🚫 Enterprise rollout without pilot
I used to think: If the pilot on one team works, it will work everywhere.
Actually: Different teams have different tech stacks, UI patterns, and skill levels. A tool that works for React may fail for Angular. Pilot on 2-3 diverse teams before committing to organisation-wide adoption.
7 Now You Try — Interview Warm-Up
Scenario: A vendor pitches an AI test generation tool claiming "zero maintenance" and "full coverage with one click." The CTO is excited and wants to sign a $150K annual contract.
What is your response as the automation architect?
Your response:
- Challenge the claim: "Zero maintenance" is impossible. All tests require review when the application changes. Ask for customer references with similar tech stacks.
- Propose a pilot: 3-month trial on one product line with measurable outcomes: time to create tests, review time per test, maintenance hours per month, and defect detection rate.
- Define exit criteria: If the pilot doesn't show 30%+ improvement in test creation time with <10% false positive rate, we don't proceed.
- Contract structure: Negotiate annual with exit clause after pilot, not multi-year commitment.
Why teams fail here
- Confusing tool adoption with strategy — buying a self-healing or AI-generation tool is a procurement decision, not a testing strategy. Without an explicit plan for who reviews AI output, how healed locators are audited, and what failure rates trigger rollback, adoption becomes chaos with a price tag.
- Anchoring ROI measurement to vendor metrics — vendors measure "tests created per hour" or "locators healed." QA architects should measure maintenance hours saved, defect escape rate change, and mean time to detect — and track them before and after. Teams that don't baseline their own numbers have no way to know if the tool is helping or just hiding problems.
- Skipping the data governance question — in regulated NZ sectors (banking, health, CoverNZ, Revenue NZ), feeding test logs or production-like data to a cloud AI vendor may breach the Privacy Act 2020 or sector-specific obligations. Most teams don't check this until after sign-off. Raise the data residency and logging question with your privacy officer before the pilot starts, not after.
- Declaring success before the first regression cycle — AI tools often look excellent for three months because they're being used on stable features. The real test is what happens when a major UI refactor hits, or when a previously passing AI-generated test needs to cover a changed business rule. Teams that declare ROI on initial uptake are frequently burned on the first full release cycle.
Key takeaway
AI doesn't reduce the need for testing judgment — it raises the stakes for it, because a confident wrong answer at scale does far more damage than a slow right one.
8 Self-Check — Can You Actually Do This?
Click each question to reveal the answer. If you got all three, you're ready to practice.
Q1. What is the realistic maintenance reduction from self-healing locators?
Vendor claims of 60-80% are optimistic. Realistic expectation for NZ teams: 30-50% reduction in locator-related failures. Self-healing fixes symptoms, not root causes like unstable UI or missing test IDs.
Q2. Why should risk-based test selection complement, not replace, full regression?
ML models can miss novel failure modes that have no historical precedent. Risk-based selection optimises PR feedback speed, but the full suite must still run before release to catch unexpected interactions.
Q3. What is the recommended pilot approach for AI tools in the NZ market?
Start with one AI capability on one product line. Measure actual results over 3 months. Document failures openly. Only consider enterprise rollout if the pilot shows measurable improvement with acceptable error rates.
9 Interview Prep — Architect Q&A
Architects evaluating AI adoption are asked about strategic fit, ROI measurement, and team capability. Here are questions you might face.
Q. "What problems can AI solve in your test strategy?"
AI excels at three problems: maintenance burden (self-healing locators reduce test updates when UIs change), scale (generate test cases from Gherkin or OpenAPI faster than humans can write them), and detection (visual regression and anomaly detection catch non-obvious bugs). AI struggles with semantic understanding: it can't judge if a visual change is a bug or an intended redesign without human curation. Map your top pain points to AI capabilities before adopting. If your problem isn't in AI's wheelhouse, you're wasting money.
Q. "How would you measure ROI on AI testing adoption?"
Don't measure only coverage %. Instead measure: 1) Maintenance hours saved per quarter, 2) Time to create a test case (before/after), 3) Mean time to detect a regression (MTTR), 4) False positive rate (AI mistakes), and 5) Team morale (do engineers trust AI output?). Real ROI = (time saved × hourly rate + defects caught × cost per escape) − (tool cost + training cost). For a NZ team, pilot on one product line for three months, track these metrics, then decide enterprise-wide rollout. If ROI isn't clear after three months, stop and reassess.
Q. "What skills do teams need for AI-driven testing?"
Traditional testers can learn AI-assisted tools, but you need architects who understand: model limitations (hallucinations, bias), prompt engineering (how to ask the agent for tests), and validation practices (reviewing generated tests critically). You also need DevOps expertise to run agents at scale, and data engineers if you're feeding production logs to models. Hire one expert as a "guide," then train the team on practices. Don't expect testers to become ML engineers; expect them to become "intelligent users" of AI tools.
Q. "What are the limitations of current AI in testing?"
Current limitations: 1) Hallucination (AI generates tests that look right but assert wrong things), 2) Black box decisions (you can't always explain why the agent chose that test), 3) Bias amplification (if your code has a bug, AI learns to validate the bug), 4) Security/privacy risks (feeding production logs to cloud AI vendors), 5) Cost at scale (per-minute APIs get expensive fast). Don't believe vendor claims of "zero maintenance" or "automatic full coverage." AI in testing is a force multiplier, not a replacement for human judgement. Organisations that win treat it as "human + AI," not "AI instead of human."