Exploratory Testing in Agile
Simultaneous learning, test design, and test execution using charters and time-boxed sessions to discover what scripted tests miss.
What it is
Exploratory testing is not random clicking. It is a disciplined, investigative approach where the tester simultaneously learns about the system, designs tests, and executes them in real time. The goal is to find what scripted tests cannot: unexpected behaviour, usability gaps, and emergent risks.
In agile teams, exploratory testing is often structured using Session-Based Test Management (SBTM):
- Charter — a mission statement defining what to test, where to look, and what risks to probe.
- Time-box — a focused session of 45–90 minutes with no interruptions.
- Debrief — a short conversation after the session to share findings, assess coverage, and decide next steps.
Because it adapts to what the tester discovers mid-session, exploratory testing excels at finding the unexpected—the behaviours nobody thought to script.
When to use it
| Occasion | Why exploratory testing helps |
|---|---|
| Throughout the sprint | Keeps testing aligned with incremental delivery rather than a big-bang phase at the end. |
| New features | When functionality is unfamiliar, exploratory testing reveals gaps in requirements and assumptions. |
| High-risk areas | Complex calculations, security boundaries, or integrations benefit from targeted deep dives. |
| After major refactoring | Smoke tests may pass, but only a human eye can spot subtle regressions in UX or logic. |
| Before release | A final sanity check to catch issues that automation and scripted tests have not covered. |
Key concepts
| Concept | Description |
|---|---|
| Test Charters | A concise mission statement (e.g., "Explore checkout flow with invalid payment methods to find error-handling gaps"). Guides the session without prescribing steps. |
| Time-boxing | Fixed-length sessions (typically 45–90 minutes) that create focus and make planning predictable. |
| SBTM | Session-Based Test Management: a structured approach to planning, tracking, and reporting exploratory testing work. |
| Debriefs | A 10-minute post-session review with the team or test lead to share findings, discuss coverage, and decide on follow-up actions. |
| Heuristics | Cognitive shortcuts (e.g., CRUD, boundaries, error handling) that guide testers toward likely defect clusters. |
Common pitfalls
| Pitfall | How to avoid it |
|---|---|
| Confusing with ad-hoc clicking | Always use charters and time-boxes. Without structure it is just guessing. |
| Failing to document findings | Use session sheets, screenshots, or video. Undocumented bugs tend to be forgotten or dismissed. |
| Sessions without charters | A charter provides mission and scope. Without one, coverage is impossible to measure. |
| Using as an excuse to skip automation | Exploratory and automated testing are complementary. Automation handles the known; exploration finds the unknown. |
| Not time-boxing | Open-ended sessions lose focus and are hard to schedule. Stick to 45–90 minutes. |
NZ context
In New Zealand's agile teams, exploratory testing should be treated as a scheduled sprint task, not an afterthought. Many NZ organisations—particularly in SaaS and fintech—run tight sprint cycles and expect testers to report progress in stand-ups. Time-boxed sessions fit naturally into this rhythm: a tester can commit to "two 60-minute exploratory sessions on the payment module" and report results the next day.
Remote-work culture is also strong in NZ. Debriefs can be async (shared session notes in Confluence or Slack) or sync (a quick Zoom call). The key is that findings are shared, not siloed.
Industry Reality
- Most teams say they do exploratory testing but mean "unscripted clicking after scripted tests pass" — charters and SBTM are rarely practised outside mature QA functions or teams that have been burnt by late defects.
- Time-boxing is almost always abandoned under sprint pressure. Sessions balloon to "whatever time is left before the demo", which destroys coverage tracking and debrief quality.
- Senior testers informally use mental heuristics (CRUD, boundaries, error paths) without naming them — the vocabulary is textbook, but the instinct is real and worth developing even if nobody calls it SBTM.
- Session notes are the first thing to get cut. In practice, many findings live only in testers' heads or in a quick Slack message. When a defect reappears two sprints later, there is no audit trail.
- In NZ SaaS teams, exploratory testing is often the only testing done on UI flows before release — automation coverage is sparse for frontend, so a tester's 60-minute session before the release branch is cut carries significant risk-gate weight.
Context guide
How the right level of Exploratory Testing in Agile effort changes based on team context.
| Context | Priority | Why |
|---|---|---|
| CoverNZ or HealthNZ sprint delivering a new citizen-facing claim or eligibility flow | Essential | Errors in eligibility or payment logic affect vulnerable people and attract ministerial scrutiny. No automation suite can anticipate every edge case in legislation-driven business rules; targeted charters on boundary conditions are the only reliable safety net. |
| NZ SaaS startup with thin automation coverage and a two-week release cadence | Essential | When the regression suite covers only happy paths, a tester's 60-minute exploratory session before the release branch is cut carries most of the risk-gate weight. Without it, unknown defects routinely reach production. |
| TransitNZ or LandNZ integrating a third-party API or map-data feed mid-sprint | High | Third-party contract boundaries are where assumption-driven test cases fail most often. Exploratory sessions on error states, timeout behaviour, and partial data returns surface integration gaps before they reach citizens. |
| Pacific Bank or Harbour Bank sprint following a major back-end refactor with stable external APIs | Medium | Regression automation handles the known contract; exploratory effort should focus on subtle UX regressions and state-transition edge cases that scripted tests did not anticipate. One or two targeted charters are sufficient alongside a strong automated run. |
| Revenue NZ or Benefits NZ compliance-verification sprint where every test step must trace to a legislative requirement | Low | When audit trails must map each test execution to a specific clause in the Tax Administration Act or the Social Security Act, freeform exploratory sessions make traceability harder to prove. Scripted test cases with documented evidence are the better primary tool; reserve exploratory time for pre-compliance smoke sessions only. |
| Pacific Air or TeleNZ team with mature automation, stable product area, and no new feature work this sprint | Low | When nothing has changed, exploratory sessions return diminishing results on well-tested ground. Invest that time in improving automation coverage or maintaining the charter library for the next high-risk sprint. |
Trade-offs
What you gain and what you give up when you adopt Exploratory Testing in Agile.
| Advantage | Disadvantage | Use instead when… |
|---|---|---|
| Discovers genuinely unknown defects that no scripted test anticipates — particularly in complex integrations and new features where assumptions are unverified. | Coverage is harder to quantify. Stakeholders accustomed to pass/fail counts find it difficult to assess progress from a session sheet versus a test-case report. | The objective is verifying a fully specified, stable behaviour — scripted or automated tests are faster, cheaper, and produce a cleaner audit trail. |
| Adapts in real time to what the tester discovers — if a thread looks promising, the session can follow it without waiting for a test-case update or re-approval cycle. | Depends heavily on the tester's skill and domain knowledge. A junior tester without heuristics or product context produces significantly lower-yield sessions than a senior with both. | The team lacks experienced testers and there is no time to develop charter-writing skill this sprint — scripted tests at least guarantee predictable, reviewable coverage. |
| Fits naturally into agile sprint cadences — time-boxed sessions map directly to sprint hours, making capacity planning straightforward and debriefs a natural stand-up input. | Session notes and debrief records require discipline to maintain. Under sprint pressure, documentation is the first casualty — without it, findings live only in the tester's memory and the risk map never gets updated. | Regulatory compliance requires every test step to be traceable to a documented requirement — exploratory sessions cannot satisfy that standard without significant additional documentation overhead. |
| Generates high-quality automation candidates — bugs found through exploration that are reproducible and high-risk are ideal targets for regression test automation, compounding the value of the session over time. | Cannot replace regression automation — re-running the same exploratory session as a regression check is inefficient, expensive, and inconsistent. Teams that conflate the two end up with neither working well. | The feature is well-understood, low-risk, and already covered by a mature regression suite — the marginal gain from an additional exploratory session is low. |
Enterprise reality
How exploratory testing in agile changes at 200–300-developer scale in NZ banks, government agencies, and telcos
- Charter generation is partially automated at this scale — tools like Xray and Zephyr integrate with Jira to propose session charters from risk-scored stories, and platforms such as Tricentis Tosca auto-suggest exploratory paths from historical defect clusters. What's still manual at a five-person startup (writing every charter from scratch) is handled by tooling that aggregates risk signals across 30+ squads and flags where manual exploration is most needed each sprint.
- Harbour Bank New Zealand's testing discipline under the NZISM (NZ Information Security Manual) and PCI DSS requires that any exploratory session touching cardholder data or authentication flows is logged with a named tester, timestamp, and debrief outcome — session sheets are not optional documentation, they are audit evidence. At small-team scale, a Slack message debrief is fine; at bank scale, the same session must produce a Confluence-linked record traceable to the PCI DSS requirement being probed.
- Coordinating exploratory coverage across 10 or more squads working on the same platform creates significant overlap risk — two squads may run charters on the same integration boundary in the same sprint with no visibility of each other's findings. Enterprise QA practices at organisations like TechServNZ and HealthNZ address this with a shared charter registry (typically a Confluence page or Xray test plan) so test leads can see at a glance which areas have been explored, which are still open, and where duplicate effort can be redirected to uncovered risk areas.
- When exploratory testing is done poorly at enterprise scale, the cost is not a single missed defect — it compounds. A Privacy Act 2020 data-exposure bug discovered post-release at a government agency can trigger an Office of the Privacy Commissioner investigation, mandatory breach notification to affected individuals, and remediation sprints that freeze new feature delivery for weeks. Organisations like Benefits NZ and Revenue NZ that handle sensitive citizen data treat the debrief record from each exploratory session as part of their breach-prevention evidence chain; losing that chain because debriefs were skipped under sprint pressure is a compliance risk, not just a quality risk.
◆ What I would do
Professional judgement — when to adopt Exploratory Testing in Agile, when to adapt it, and what to watch for.
The bottom line: Exploratory testing is most powerful when it runs early enough for findings to be fixed — schedule charters in the first half of the sprint, debrief every session no matter how short, and never let "we'll explore at the end" become the plan.
Best Practices
- ✓ Write the charter before the session starts — even one sentence ("Explore the invoice PDF export to find rendering and data-accuracy issues") gives the session a testable mission and makes debriefs meaningful.
- ✓ Use a timer. Set 60 or 90 minutes, then stop. If the area needs more coverage, write a new charter for a follow-up session rather than extending indefinitely.
- ✓ Record your screen during sessions. Video is faster to capture than notes and gives developers an exact reproduction path, which cuts triage time in half.
- ✓ Pair with a developer during high-risk sessions — they see code-level cues you don't, and you see UX gaps they ignore. Ten minutes of pairing often surfaces a defect that neither person would find alone.
- ✓ Keep a personal heuristic list. Over time, note which CRUD operations, boundary conditions, or error states you keep finding bugs in — these become your highest-yield charter starting points on new features.
- ✓ Log findings as you go, not at the end. Memory degrades fast during exploratory work; capture observations immediately in a running note doc, Jira comment, or session sheet.
- ✓ Debrief every session — even a 5-minute async Slack update counts. Share what you tested, what you found, what you skipped, and what you'd charter next. This keeps the team's risk picture current.
- ✓ Feed automation from exploration. When you find a reproducible bug through exploratory testing, assess whether the test scenario should be added to the regression suite — exploration that improves automation coverage has compounding value.
Common Misconceptions
❌ Myth: Exploratory testing is just testing without a plan — any tester can do it by instinct.
Reality: Effective exploratory testing requires deliberate structure: a charter defines what to investigate, a time-box maintains focus, and a debrief captures coverage. Without these, you have ad-hoc clicking — not exploratory testing. Skill compounds over time as testers build a library of heuristics and risk instincts, but that is the opposite of "anyone can do it without preparation".
❌ Myth: Exploratory testing replaces automated regression testing, so you can invest in one or the other.
Reality: They serve fundamentally different purposes. Automation reliably re-checks known behaviour at speed — it cannot discover what nobody thought to test. Exploratory testing discovers unknown risks and novel failure modes — it cannot run 800 regression scenarios in five minutes. Teams that drop automation in favour of exploration get slow, expensive regression coverage. Teams that drop exploration in favour of automation ship bugs that no test case ever anticipated.
❌ Myth: Exploratory testing is untrackable and doesn't fit into agile sprint reporting.
Reality: SBTM was specifically designed to make exploratory testing reportable. Time-boxed sessions map directly to story points or hours in a sprint plan. Session sheets capture coverage areas and bug counts. A tester can commit to "three 60-minute sessions on the checkout module this sprint" and report against that commitment in stand-ups — the same as any other sprint task.
Career level guidance
| Level | Focus | What to practise |
|---|---|---|
| Junior | Structure & discipline | Run chartered sessions within a time-box, take thorough notes, report findings clearly, and learn common heuristics (boundaries, CRUD, error handling). |
| Senior | Risk targeting & coaching | Design charters based on risk analysis, facilitate debriefs, pair with juniors during sessions, and identify automation candidates from exploratory findings. |
| Test Lead | Strategy & metrics | Balance exploratory and automated testing across the release, track session coverage vs risk, coach the team on SBTM, and report exploratory findings to stakeholders in business terms. |
Senior engineer insight
Teams that do exploratory testing well treat charters as a contract with themselves: specific enough to guide a session, loose enough to follow an interesting thread when one appears. The pattern that consistently separates high performers is the post-session debrief — even a five-minute async Slack update forces the tester to articulate what they found, what they skipped, and what they would charter next. That discipline compounds across sprints into genuine risk coverage. Teams that struggle skip debriefs under time pressure, which means findings live only in one person's head and risk maps never get updated.
The most common mistake: treating exploratory testing as something to squeeze in after scripted tests pass, rather than scheduling it as a first-class sprint task with estimated time and a defined charter. By the time it gets started, there are 40 minutes left before the demo and the session produces one bug that gets ignored because there is no time to triage it.
From the field
A Wellington fintech team running two-week sprints assumed their exploratory testing was solid — they had a dedicated QA engineer who spent the last day of every sprint doing "free testing" before the release branch was cut. What they discovered mid-year was that every session was covering the same happy paths because there were no charters: the tester defaulted to whatever felt familiar under time pressure. After a payment rounding defect slipped to production — one that a targeted charter on boundary arithmetic would have caught in 20 minutes — the team adopted SBTM with explicit charters written during sprint planning. Charter topics were tied directly to the sprint's risk register: new integrations, changed business logic, and areas with no automation coverage. Within two sprints, bugs found per session tripled and, more importantly, the team could point to specific charters in sprint reviews as evidence of what had been tested. The lesson that travels: exploratory testing without charters is not a strategy, it is a habit — and habits default to comfort rather than risk.
Why teams fail here
- Scheduling exploratory sessions as leftover time at the end of the sprint — when pressure is highest and focus is lowest — means sessions are rushed, charters are skipped, and findings are rarely actioned before the release branch is cut.
- Writing charters that are too broad ("explore the whole checkout flow") rather than targeted ("explore checkout with a cart containing a mix of GST-exempt and standard-rated items") results in sessions that skim the surface and find nothing of consequence.
- Abandoning debriefs under sprint pressure destroys the team's ability to track coverage across sessions — without debrief records, no one knows which charters were run, which areas were skipped, and whether the risk picture has actually improved.
- Treating exploratory testing as a replacement for automation rather than a complement means the team ends up manually re-checking known regressions every sprint instead of using that session time to discover what automation cannot.
Key takeaway
Exploratory testing done well is not unstructured — it is structured curiosity: a targeted charter gives you a mission, a time-box keeps you honest, and a debrief turns a single session into shared team knowledge that makes every subsequent sprint safer.
How this has changed
The field moved. Here is how Exploratory Testing in Agile evolved from its origins to current practice.
Agile teams initially resist scripted testing — it feels like waterfall documentation. Exploratory testing fits agile because it requires no upfront documentation and delivers immediate feedback. SBTM provides the structure that makes it accountable.
Elisabeth Hendrickson publishes "Better Software Through Exploration." The community articulates how exploration complements automation — automation handles regression, exploration finds what automation cannot.
The "whole team quality" concept moves exploration from a phase to a continuous practice. Testers explore during development, not after. Bug hunts and story kick-off explorations become standard agile ceremonies.
Risk-based exploration charters integrate into sprint planning. Testers identify high-risk areas in new features and allocate charter-guided sessions to them before sprint review.
AI tools propose charters from risk analysis and user story content. Automated testing handles repetitive verification; human exploration focuses on creative, adversarial, and edge-case investigation — especially for AI-generated systems.
Self-Check
Click each question to reveal the answer.
Q1: What is the difference between exploratory testing and ad-hoc testing?
Ad-hoc testing is unstructured, undocumented clicking with no deliberate mission. Exploratory testing uses a charter to define what to investigate, a time-box to maintain focus, and a debrief to share findings. Both are unscripted in the moment, but only exploratory testing is disciplined and repeatable.
Q2: What three elements make up a Session-Based Test Management (SBTM) session?
A charter (the mission — what to test and what risks to probe), a time-box (a focused 45–90 minute block with no interruptions), and a debrief (a short post-session review to share findings, assess coverage, and decide on follow-up charters).
Q3: Why does exploratory testing complement rather than replace automated regression testing?
Automation reliably re-checks known behaviour at speed but cannot discover what nobody thought to test. Exploratory testing finds novel failure modes, usability gaps, and emergent risks that no test case anticipated, but cannot run hundreds of regression scenarios in minutes. Both are needed: automation covers the known, exploration finds the unknown.
Q4: What is a test charter, and what should a well-written one contain?
A charter is a concise mission statement for a session. A good one names the area to explore, the approach or angle, and the risk or question being investigated — for example: "Explore the payment gateway integration with invalid card numbers to find error-handling and timeout behaviour." It guides the session without prescribing every step.
Q5: What are heuristics and why do experienced exploratory testers rely on them?
Heuristics are cognitive shortcuts — patterns like CRUD operations, boundary conditions, error handling, and state transitions — that point testers toward areas where defects are most likely to hide. Experienced testers build a personal library of heuristics over time, allowing them to write higher-yield charters and focus sessions on the areas with the highest risk.
Q6: Your team is running a two-week sprint on TransitNZ's online vehicle licensing portal. Automation covers the happy-path renewal flow, but the sprint added a new edge case: multi-vehicle renewals where one vehicle has a Warrant of Fitness that expired during the cart checkout. You have 90 minutes before the sprint demo. How do you write your charter and focus your session?
Write a targeted charter such as: "Explore multi-vehicle renewal checkout when one vehicle's WoF expires mid-session — focus on error messaging, cart state, and payment outcome." Spend the first 15 minutes mapping the flow on paper, then systematically trigger the WoF-expiry condition at different checkout stages (before payment, during payment, after payment confirmation). Capture screenshots of every error state and note whether messaging is clear to a citizen without technical knowledge. Skip edge cases already covered by automation — use your 90 minutes for the novel, high-risk paths that no existing test covers.
Q7: What is the key difference between exploratory testing and risk-based testing, and when would you use each?
Risk-based testing is a prioritisation strategy — it ranks features and test efforts by likelihood and impact of failure, and is done during planning. Exploratory testing is an execution technique — it is how you investigate an area in real time using charters and time-boxes. They are complementary: risk-based analysis tells you where to focus; exploratory testing is how you investigate those high-risk areas thoroughly. In practice, a risk analysis informs which charters you write, so both run together rather than as alternatives.
Q8: A developer says "we don't need to write charters — the best exploratory testers just know where to look by instinct." What is wrong with this view and how do you respond?
Instinct is real and valuable, but without a charter there is no coverage record, no debrief anchor, and no way for a team lead or stakeholder to know what was and was not tested. When a defect surfaces post-release, undocumented sessions leave no audit trail. Charters also help less-experienced testers develop instinct deliberately — they make the mental model explicit. You can respectfully push back: "I agree good testers develop strong instincts, but a one-sentence charter takes two minutes and makes our session reportable in the sprint review and defensible if something ships broken."
Q9: An interviewer asks when you would advise a team NOT to use structured exploratory testing. What situations would you name?
Exploratory testing is a poor fit when the test objective is to verify a known, repeatable specification — scripted or automated tests are more efficient and auditable there. It is also less suitable for compliance testing where every step must be documented against a standard (such as Revenue NZ audit trails or CoverNZ privacy obligations), because the freeform nature makes traceability harder to prove. Finally, if the system is so unstable that it crashes within minutes, a charter-driven session is wasted — stabilise the environment first. Exploratory testing shines on novel features, complex integrations, and usability risks; it is not the right tool when repeatability and evidence trails are the primary concern.