Agentic AI & The Future of QE
Transition from volume-based automation to intelligence-based assurance. Learn how to orchestrate multi-agent AI teams and leverage production observability as your primary quality signal.
1 The Hook — Why This Matters
By 2026, the era of the "scripting engineer" is over. Traditional automation, with its brittle selectors and manual maintenance, cannot keep pace with AI-generated codebases that deploy dozens of times per day. A leading NZ financial institution recently discovered that 40% of their regression suite was testing features that had been deprecated or fundamentally changed by AI-driven experiments.
The future of quality is not more tests; it is more intelligence. Agentic AI systems don't just "replay" actions; they understand context, self-heal, and autonomously decide what to verify based on production risk. If you are still managing scripts instead of agents, you are the bottleneck.
2 The Rule — The One-Sentence Version
Shift from orchestrating scripts to orchestrating agents; replace static thresholds with adaptive, observability-driven quality signals.
In 2026, a "test failure" is a data point in a broader observability mesh. The architect's job is to ensure that agents have the right telemetry (OpenTelemetry) to make autonomous decisions without human intervention.
3 The Analogy — Think Of It Like...
From a Librarian to a Symphony Conductor.
A traditional automation engineer is like a librarian: they categorise, store, and retrieve static documents (scripts). An Agentic AI Architect is like a conductor: they don't play every instrument, but they ensure that the different agents (authoring, execution, analysis) work in harmony to produce a coherent result (Quality). You don't manage the notes; you manage the performance.
Senior engineer insight
The single biggest shift in how I think about agentic testing isn't the AI itself — it's accepting that non-determinism is a feature, not a bug. When we deployed an LLM-driven test authoring agent at an NZ bank, it found an edge case in the Revenue NZ integration that 18 months of scripted regression had never caught, precisely because it reasoned about the data rather than replaying fixed steps. That humbled every senior engineer on the team.
Most common mistake: treating AI-generated tests as "good enough to ship" without a human reviewing whether the assertion actually validates business intent, not just UI state.
From the field
A Wellington-based payments team assumed their self-healing locator tool would "just work" across their checkout flow after a major React upgrade. It did — the locators healed correctly on every run. What nobody caught for three weeks was that the AI had silently decided to skip a step in the payment confirmation sequence because it looked visually redundant. No test failed. The agent was confidently confirming a broken flow. The incident made it onto the retrospective slide deck as "the time the robot lied to us". The lesson that generalises: self-healing agents need assertion contracts defined separately from locator strategy — healing the path to a button and asserting the business outcome after clicking it are two different responsibilities, and conflating them is how silent regressions stay hidden.
4 Watch Me Do It — Step by Step
Implementing an Agentic Quality Mesh follows these modern patterns.
- Implement OpenTelemetry (OTel) Foundations Before agents can be smart, they need data. Ensure every service emits standard traces and logs. This is the "Universal Language" your AI agents will use to understand system state.
- Deploy "Sentinel Agents" for Production Monitoring Instead of static probes, use agents that monitor real-user sessions and autonomously generate "Shadow Tests" in the background to verify edge cases found in the wild.
- Integrate Self-Healing Authoring Agents Configure your framework to use LLM-based locators that can survive 80% of DOM changes by understanding the intent of the element (e.g., "The Buy Button") rather than its ID.
- Establish "AI Governance Gates" As AI writes more of your tests, you need automated validators to ensure the tests themselves aren't hallucinating or missing critical security checks.
| Capability | Traditional (2023) | Agentic (2026) |
|---|---|---|
| Test Creation | Manual scripting | Autonomous generation from specs/PRs |
| Maintenance | Manual locator updates | Self-healing via contextual intent |
| Risk Analysis | Human intuition | Predictive models based on production telemetry |
| Failures | Stack traces & screenshots | Correlated distributed traces & root-cause analysis |
5 When to Use It / When NOT to Use It
✅ Adopt Agentic AI when...
- Deploying multiple times per hour
- Managing complex microservice meshes
- Testing AI-infused product features
- UI changes are frequent but logic is stable
❌ Stick to Traditional when...
- Regulated systems requiring fixed, auditable scripts
- Static, low-change applications
- Small projects where agent setup exceeds ROI
- Safety-critical systems with zero tolerance for non-determinism
6 Common Mistakes — Don't Do This
🚫 "Black Box" Agent Adoption
I used to think: The AI is smart enough to handle everything on its own.
Actually: Without clear boundaries and human oversight, agents can create "Ghost Pass" scenarios where tests pass but critical business logic is actually broken. Always implement Observability Observability — monitor your monitors.
🚫 Ignoring Data Sovereignty in AI
I used to think: Just feed the logs into the LLM for analysis.
Actually: In New Zealand, Māori Data Sovereignty and the Privacy Act 2020 require strict control over where data is sent. Using offshore AI agents for root-cause analysis can violate these mandates. Architects must design Hybrid Agent Models that keep sensitive data onshore.
7 Now You Try — Strategy Session
Scenario: You are the Architect at a mid-sized NZ Fintech. Your CEO wants to move from bi-weekly releases to on-demand deployments. Your current automation takes 4 hours to run and is 15% flaky.
What is your three-step plan to use Agentic AI to enable on-demand deployment?
- Observability First: Implement OpenTelemetry to correlate test failures with backend logs, immediately identifying if a failure is a code bug or an environment flake.
- Risk-Based Selection: Deploy an agent that analyzes the diff of each PR and selects only the 5% of tests impacted by the change, reducing run time from 4 hours to 5 minutes.
- Self-Healing Locators: Implement an AI-driven locator strategy to eliminate the 15% flakiness caused by UI drift, ensuring the pipeline only blocks on real regressions.
Why teams fail here
- Hallucination blindness: LLM-generated tests pass code review because they look syntactically correct, but they assert the wrong business rule — the AI inferred intent from naming conventions, not actual requirements. TransitNZ teams discovered this when an auto-generated test "verified" a fee calculation by checking it was a number, not that it matched the RUC schedule.
- Governance vacuum: The agent deployment outruns the governance model. Teams go from zero agentic testing to full autonomous generation without defining who owns a test the AI wrote, how it gets reviewed, or what happens when it fails in production at 2am.
- Context starvation: Agents are fed raw code and DOM snapshots but denied access to user stories, known defect history, or domain rules — so they optimise for coverage breadth over risk depth. A Sentinel agent running against an CoverNZ claims portal with no knowledge of the eligibility rules will produce technically comprehensive but practically useless tests.
- Offshore data leakage: Teams pipe production trace data directly into external LLM APIs for root-cause analysis, not realising they've just sent real user session data offshore in breach of the Privacy Act 2020. NZ health sector and council systems are especially exposed — the analysis capability is compelling, but it needs a hybrid on-premises agent model, not a raw API call to a US endpoint.
Key takeaway
The architect who wins with agentic AI isn't the one who automates the most — it's the one who defines the guardrails precisely enough that the agents know exactly when to stop and ask a human.
8 Self-Check — The Architect's Verdict
Q1. What is the shift from "Volume-based" to "Intelligence-based" assurance?
Moving away from simply counting test cases (coverage) to measuring risk mitigation and confidence through correlated data from production and testing.
Q2. Why is OpenTelemetry critical for Agentic AI in QE?
It provides a universal, standardized data stream that AI agents can use to understand the state of distributed systems, enabling them to diagnose root causes autonomously.
Q3. What is the role of a human Architect in an Agentic Mesh?
Governance, context engineering, strategy, and judgment. The Architect defines the quality objectives and ensures the agents operate within ethical and safety guardrails.
9 Interview Prep — Architect Q&A
Senior and architect-level interviewers ask about AI agent design philosophy, risk mitigation, and system thinking. Here are questions you might encounter.
Q. "How would you design an AI test agent for a large system?"
Start with clear guardrails: define the agent's scope (what systems it can test), assert on business outcomes rather than implementation details, and build observability so you can trace decisions. Use context engineering—give the agent access to requirement documents, user stories, and production logs. Test the agent's test generation like you'd test any code: with integration tests and human review before it runs unsupervised. Design for graceful degradation: if the agent generates a bad test, it should fail validation, not ship to production.
Q. "What are the risks of AI-driven testing?"
The biggest risk is hallucination: an AI-generated test that looks correct but asserts the wrong behaviour. This can hide real bugs. Mitigation: never skip human review for AI-generated tests. Second risk is bias amplification—if your production code has a bug, the AI learns from it and generates tests that validate the buggy behaviour. Counter this with explicit "known issues" documentation that informs the agent. Third: governance. Without clear oversight, AI agents can drift from intended scope. Implement automated validators that catch policy violations before execution.
Q. "How do you handle hallucinations or false positives in AI testing?"
Hallucinations appear as tests that pass consistently but assert incorrect logic. Catch them with peer review of generated tests (ask: "Does this test make sense?"), strong assertion clarity (avoid vague assertions like "check if page loads"), and exploratory testing by humans to catch edge cases the agent missed. For false positives, implement two-phase validation: the agent runs tests in sandboxed environments first, and only critical tests promoted to the main pipeline. Log every agent decision so you can audit when things go wrong and retrain the agent model.
Q. "What's the difference between agentic and traditional automation?"
Traditional automation is deterministic: a test is written once and replayed identically. Agentic automation is adaptive: the agent evaluates context (diffs, logs, user behaviour) and makes decisions about what to test and how. Traditional suits static applications and regulated industries where reproducibility is critical. Agentic suits rapid-release, AI-driven products where the application changes faster than manual test updates. The hybrid approach: use agents to generate tests, but treat those tests as first-class code that humans review and maintain.