Pairwise Testing
A combinatorial technique that ensures every pair of input values is tested at least once — typically cutting a combinatorial explosion of test cases by 80–95% while still catching the majority of defects.
1 The Hook
A Wellington logistics firm rebuilt its parcel-booking screen. It had four dropdowns: carrier (PostNZ, CourierPost, Aramex, NZ Couriers), service level (standard, overnight, rural, Saturday), package size (satchel, small box, large box, pallet), and payment (account, credit card, on-account credit). The team tested the combinations they used most often — CourierPost overnight small box on account, PostNZ standard satchel credit card — and everything looked clean. Shipped on a Friday.
On Monday the support queue lit up. Rural overnight bookings were silently dropping the rural surcharge, but only when paid by credit card. Nobody had tried that exact combination, because each dropdown worked perfectly on its own. The carrier was fine. The service level was fine. The payment method was fine. The bug lived in the pair — rural service crossed with credit-card payment — and the team would have needed all 256 combinations to be sure of catching it by brute force, which no one has time for in a sprint.
This is the trap pairwise testing is built for. Most interaction bugs are caused by two settings clashing, not by some rare four-way alignment. You do not need every combination. You need every pair, and that is a far smaller, achievable set.
The tool is the easy part. After 20-odd years I can say the failure mode I see most is teams feeding the wrong parameters into PICT and getting a beautifully generated test set that covers combinations nobody can actually trigger. You have to model constraints first. If Afterpay disables Click & Collect in your pricing engine, those pairs are invalid — but PICT will cheerfully generate them and your testers will spend hours trying to reproduce a state the system never allows. Worse, the valid-but-risky pairs get diluted. Before you open any tool, draw a dependency map: which parameters genuinely interact in shared code? Which combinations are blocked by business rules? Add the constraints to your PICT input file. The 20 minutes spent on that step saves three days of chasing phantom bugs.
2 The Rule
Most defects are triggered by one or two interacting inputs — so instead of testing every combination, design the smallest set of cases that pairs every value of every parameter with every value of every other parameter at least once.
3 The Analogy
Seating a wedding so every family group meets every other.
A couple wants every side of the family to have met every other side by the end of a Marlborough wedding. They could throw one giant dinner where all 256 guests share one table — complete coverage, totally impractical. Or they could rotate people across a handful of tables so that every pair of families ends up sitting together at some point during the night. The clashes that matter — Aunty A who cannot stand Uncle B — show up between two people, not among five. Cover every pair and you will surface every feud that is going to happen.
Pairwise testing is that seating plan. You do not need every guest in the room at once; you need every pair to have shared a table. A small, well-designed rotation gets you there.
What it is
Pairwise testing (also called all-pairs testing) is a combinatorial test design technique. Rather than testing every possible combination of input values, you select a minimal set of test cases that covers every pair of values at least once. The insight is that most defects are caused by interactions between one or two variables — not by complex interactions among many variables simultaneously.
This is distinct from equivalence partitioning or boundary value analysis, which test individual inputs in isolation. Pairwise is for when you have multiple inputs that could interact with each other, and you need to balance coverage against the cost of running many tests.
ISTQB definition: Pairwise testing is a black-box test design technique in which test cases are designed to cover all pairs of values of two parameters at least once. It is a specific form of combinatorial testing (CTFL 4.0 section 4.2).
The problem it solves
Imagine testing a checkout form with four configuration parameters, each with four possible values:
- Payment method: Credit card, POLi, Afterpay, Gift card
- Delivery type: Standard, Express, Same-day, Click & Collect
- Customer type: Guest, Registered, VIP, Business
- Discount code: None, 10% off, Free shipping, Bundle deal
Full combinatorial coverage: 4 × 4 × 4 × 4 = 256 test cases. Running 256 test cases for a single feature is rarely realistic in a sprint.
Pairwise testing covers all two-way interactions (every pair of values tested together at least once) in approximately 20–25 test cases. That is the same defect-catching power for combinations, at less than 10% of the testing cost.
When to use it
- Testing systems with many configuration options where combinations matter (software settings, checkout options, filter combinations)
- Form fields with multiple valid values that interact (discount code + payment method + delivery type)
- Feature flags — when multiple flags can be on or off, pairwise covers the interactions without testing every permutation
- Compatibility matrices — OS × browser × screen resolution
- Regression suites where scope needs to be cut without eliminating coverage of interactions
NZ examples where pairwise applies
- NZ e-commerce checkout: payment method (credit card / POLi / Afterpay) × delivery type (PostNZ standard / CourierPost / Click & Collect) × discount code (none / percentage / free shipping)
- NZ banking: account type (everyday / savings / term deposit) × transaction type (transfer / bill payment / international) × amount range (small / medium / large)
- Software configuration screen: browser (Chrome / Firefox / Safari / Edge) × OS (Windows / macOS / iOS / Android) × locale (en-NZ / mi-NZ)
Worked NZ example
A NZ online booking system has three parameters:
- Payment method: Credit card, POLi, Afterpay
- Delivery type: Standard (3–5 days), Express (next day), Click & Collect
- Location: Auckland, Wellington, Christchurch
Full combinations: 3 × 3 × 3 = 27 test cases
Pairwise covers all pairs in 9 test cases:
| Test # | Payment method | Delivery type | Location | Pairs covered |
|---|---|---|---|---|
| 1 | Credit card | Standard | Auckland | CC+Std, CC+AKL, Std+AKL |
| 2 | Credit card | Express | Wellington | CC+Exp, CC+WLG, Exp+WLG |
| 3 | Credit card | Click & Collect | Christchurch | CC+C&C, CC+CHC, C&C+CHC |
| 4 | POLi | Standard | Wellington | POLi+Std, POLi+WLG, Std+WLG |
| 5 | POLi | Express | Christchurch | POLi+Exp, POLi+CHC, Exp+CHC |
| 6 | POLi | Click & Collect | Auckland | POLi+C&C, POLi+AKL, C&C+AKL |
| 7 | Afterpay | Standard | Christchurch | AP+Std, AP+CHC, Std+CHC |
| 8 | Afterpay | Express | Auckland | AP+Exp, AP+AKL, Exp+AKL |
| 9 | Afterpay | Click & Collect | Wellington | AP+C&C, AP+WLG, C&C+WLG |
Every pair is covered at least once across these 9 tests. If Afterpay + Click & Collect has a bug (a known issue with some NZ payment providers and in-store pickup), test #9 will catch it. If Express delivery to Christchurch has a routing issue, test #5 will catch it.
Important: do not build pairwise tables by hand for more than 3–4 parameters. The algorithm is complex and humans make pairing errors. Use a tool (PICT, AllPairs, Hexawise) to generate the table from your parameter list.
Why it works
Research by NIST (National Institute of Standards and Technology) analysed hundreds of software defects and found that:
- ~70% of defects are caused by a single parameter (covered by equivalence partitioning)
- ~90% of defects are caused by interactions between at most 2 parameters (covered by pairwise testing)
- ~98% of defects are caused by interactions between at most 3 parameters (covered by 3-way combinatorial testing)
This means pairwise testing — covering all 2-way interactions — catches approximately 90% of the defects that full combinatorial testing would catch, at a fraction of the test count. The remaining ~10% require 3-way or higher-order coverage and are typically reserved for safety-critical systems.
Tools
- PICT (Pairwise Independent Combinatorial Testing, Microsoft, free CLI tool) — the most widely used pairwise generator. You define your parameters and values in a text file; PICT outputs the minimum test set. Runs on Windows, macOS, and Linux.
- AllPairs (free, Python-based) — similar to PICT; useful in Python-heavy automation environments
- Hexawise (commercial, web-based) — more user-friendly interface; produces pairwise and higher-order combinatorial sets; good for larger parameter sets
Using PICT: a quick example
Create a .txt file with your parameters and values:
| File content |
|---|
Payment: Credit card, POLi, Afterpay |
Delivery: Standard, Express, Click & Collect |
Location: Auckland, Wellington, Christchurch |
Run pict booking.txt and PICT outputs the 9-row test set above. For 10 parameters with 5 values each (full combinations: 5^10 = ~10 million), PICT generates approximately 35 test cases.
When not to use it
- Safety-critical systems — medical devices, aviation, nuclear control systems may require full combinatorial testing or even higher-order coverage. Pairwise is not sufficient when all combinations must be verified for safety.
- Known 3-way (or higher) interactions — if you already know that three specific parameters interact in a way that causes defects, use 3-way combinatorial testing (or targeted decision table testing) rather than pairwise.
- Very few parameters or values — if you have only 2 parameters with 3 values each, full combinatorial is only 9 tests. There’s no need for pairwise reduction.
Bugs pairwise testing typically catches
- A discount code that fails only when combined with Afterpay as the payment method (not with credit card or POLi)
- Click & Collect orders that fail for Wellington customers but not Auckland or Christchurch
- A form that only breaks when a long company name is entered and the city is Christchurch (due to a character limit in the Christchurch address database lookup)
- Express delivery unavailable for certain NZ regions, but only when a specific discount code is applied (the two interact in the pricing engine)
- A configuration screen where two specific feature flags enabled simultaneously cause a conflict, even though each flag works correctly in isolation
ISTQB mapping
| Syllabus ref | Topic | Level |
|---|---|---|
| CTFL 4.0 — 4.2 | Combinatorial testing techniques — pairwise as a specific technique | Foundation |
| FL-4.2 K3 | Apply pairwise testing to derive test cases for a given component or system | Foundation LO |
| CTAL-TA 3.2 | Advanced combinatorial techniques, n-way testing, constraint modelling | Advanced / Senior |
The ISTQB Foundation exam expects you to apply pairwise testing (K3). You must be able to identify parameters and values from a specification, understand what a pairwise table covers, and know when the technique is appropriate.
Tips
Don’t build pairwise tables by hand — use PICT or AllPairs. Feed in your parameters and values, and the tool generates the minimum set in seconds. Pairwise is especially valuable for regression suites where you need to cut scope without cutting coverage of interactions. If your sprint regression suite has 200 tests covering a configuration-heavy feature, a well-designed pairwise set might achieve the same interaction coverage in 30.
- Identify parameters carefully — the quality of your pairwise set depends entirely on correctly identifying the parameters that can interact. Think about which inputs go through the same code path together.
- Add a few targeted tests beyond pairwise — if you have business knowledge that suggests a particular 3-way interaction is risky, add a specific test for it. Pairwise is a baseline, not a ceiling.
- Use pairwise for compatibility matrices — if you need to test across browsers, operating systems, and screen sizes, pairwise dramatically reduces the matrix size without abandoning cross-environment coverage.
- Combine with risk-based prioritisation — not all pairs are equally risky. Use business knowledge to flag which parameter combinations are highest-risk and verify those are included in your generated set.
Practice this technique: Try Junior Practice 06 — Dropdown & select bugs.
4 Industry Reality
- Requirements rarely list parameters neatly. In real projects you get a Confluence page or a Jira ticket that describes the feature in prose. Identifying the independent parameters — and their discrete values — is skilled analytical work that takes time, and product owners often cannot tell you whether two config options truly interact or not.
- Tools are available but seldom set up for you. PICT is free but needs to be installed and run from the command line; most organisations do not have it in their standard toolchain. Junior testers often end up building tables by hand in Excel, which is where the pairing mistakes creep in. Senior testers fight to get PICT or Hexawise into the pipeline once, then document the workflow so the whole team can reuse it.
- The hardest part is knowing when to stop. Stakeholders will ask “have you tested all combinations?” Explaining why 20 test cases cover 90% of interaction defects is a communication task, not a testing task. Good testers prepare a one-page justification they can share with the PM or tech lead before the review meeting.
- Legacy systems often have undocumented interactions. You may inherit a system where three parameters have been quietly coupled in the code for years. Pairwise surfaces these, but the fix may be expensive. Expect pushback when a pairwise run finds a bug nobody knew existed in a five-year-old integration.
- NZ context: small team sizes mean pairwise is even more valuable. A five-person QA team at an Auckland SaaS company cannot run 300 manual tests per sprint. Pairwise is one of the techniques that makes realistic coverage possible at NZ scale without moving to full automation immediately.
5 When to Use It — and When Not To
✓ Use it when
- You have 3 or more independent parameters, each with 2 or more values, that pass through shared code paths — especially pricing engines, routing logic, or configuration screens
- Full combinatorial coverage would produce more than ~30 test cases and the sprint budget cannot support it
- You are cutting an existing regression suite and need to preserve interaction coverage while reducing case count
- You are testing a compatibility matrix (browser × OS × locale, or device × network × language) where every full combination is impractical
- You have no specific knowledge of higher-order (3-way+) interactions and want a principled, defensible baseline
✗ Skip it when
- You have only 2 parameters — pairwise and full combinatorial are identical; there is no reduction to gain
- The full combinatorial count is already small (under 20 cases) — just run them all and get certainty
- The system is safety-critical (medical device firmware, aviation control, SCADA) where regulators or standards (IEC 62304, DO-178C) require exhaustive or higher-order combinatorial testing
- You already have documented evidence of a specific 3-way interaction causing defects — use 3-way combinatorial or a targeted decision table instead
- Parameters are not independent — if selecting Afterpay already disables Click & Collect (a constraint), you need constraint modelling in PICT, not a naive pairwise run
Context guide
How the right level of pairwise testing effort changes based on project context.
| Context | Priority | Why |
|---|---|---|
| CoverNZ online claims portal — injury type, claimant status, lodgement channel, and document type each with 3–4 values | Essential | Incorrect entitlement calculations depend on specific combinations of claim type and lodgement path. Full combinatorial would be 200+ cases per sprint; pairwise gives defensible coverage in under 25. Privacy Act 2020 obligations mean data-handling paths must be verified across combinations, not just in isolation. |
| TransitNZ TransitNZ online licence renewal — browser, OS, licence type, and RealMe authentication status | Essential | Cross-browser and cross-device bugs on government portals affect citizen access. A pairwise set for 4×4×3×2 parameters reduces 96 combinations to roughly 16–20, keeping the regression suite inside a realistic sprint budget while preserving interaction coverage. |
| Harbour Bank or Pacific Bank internet banking — account type, transaction type, amount tier, and authentication method | High | Pricing and routing logic for bill payments and international transfers interacts across account type and amount. Pairwise surfaces combination bugs (e.g. term deposit + international + high-value) that single-parameter tests never reach, without requiring the full matrix. |
| Benefits NZ (Work and Income) Flexi-wage or benefit application — employment type, region, subsidy tier, and channel | High | Government entitlement calculations are subject to audit; a missed combination bug can cause incorrect payments to hundreds of recipients. Pairwise with risk-based augmentation (add targeted cases for highest-volume combinations) is the appropriate approach at Benefits NZ scale. |
| Pacific Air or TeleNZ product configuration — plan type, add-ons, billing cycle, and customer tier | Medium | Pricing-engine interactions are real but recoverable — a billing error is noticeable and fixable. Pairwise is appropriate for each release but the urgency is lower than safety-critical or regulatory paths. Skip it only when the combination space is already small enough to run fully. |
| Internal admin tool with 2 parameters and 3 values each (18 full combinations) | Low | When full combinatorial coverage fits in a sprint budget, just run it all — pairwise adds process overhead without reducing test count enough to matter. Reserve pairwise for situations where the full matrix is genuinely impractical. |
Trade-offs
What you gain and what you give up when you choose pairwise testing.
| Advantage | Disadvantage | Use instead when… |
|---|---|---|
| Reduces a 100–500 case combinatorial set to 15–30 cases while retaining two-way interaction coverage — the category responsible for roughly 90% of combination defects | Three-way (and higher) interaction bugs are invisible to pairwise. A defect that only surfaces when three specific values co-occur — e.g. rural + Afterpay + same-day delivery — will not be caught unless you add targeted tests or escalate to 3-way combinatorial | You have a documented three-way interaction already on the risk register — use 3-way combinatorial via PICT /o:3 or add a targeted test for that specific triple |
| Generates a mathematically verified, defensible coverage rationale you can present to a PM, tech lead, or auditor — far stronger than “we tested the combinations we thought of” | The technique only covers the parameters you identify. If you model UI fields instead of the underlying logic variables, the generated set is mathematically correct but practically useless — garbage-in, garbage-out applies fully here | You cannot confidently enumerate the independent parameters — consider cause-effect graphing to model the parameter dependencies explicitly before choosing a combinatorial strategy |
| Reusable: the PICT input file is a version-controlled test asset. Adding a new value (e.g. a new payment method) means updating one file and regenerating — the set stays minimal and complete | Requires constraint modelling when parameters are not fully independent. If certain combinations are invalid (Afterpay unavailable for Click & Collect), failing to encode that in PICT produces invalid test cases that waste tester time and pollute results | The full combinatorial count is already under 20 — just run everything and get certainty rather than managing a pairwise generator for minimal gain |
| Scales to large parameter spaces (10+ parameters, 5+ values each) where full combinatorial is millions of cases — the reduction factor grows dramatically as the parameter space grows | Does not specify expected outputs per combination — you still need domain knowledge or a specification to know what the right result should be for each generated test case. Pairwise tells you which cases to run, not what to assert | You have well-defined per-combination expected outputs and a small set of binary conditions — a decision table gives you explicit outcome mapping that pairwise does not |
Enterprise reality
How pairwise testing changes at 200–300-developer scale in NZ enterprise
- Pairwise generation is fully automated — teams at CloudBooks and ListRight run tools like Hexawise or PICT inside CI pipelines so engineers never hand-craft parameter tables; the tool outputs a test matrix and feeds it directly into parameterised test frameworks such as JUnit or pytest.
- Compliance mandates the coverage artefact, not just the tests — under the NZ Information Security Manual (NZISM) and the Privacy Act 2020, organisations like Revenue NZ and HealthNZ must demonstrate that test coverage decisions are documented and risk-justified; a pairwise coverage report satisfies that audit trail in a way that ad-hoc exploratory testing cannot.
- Tooling at volume means dedicated combinatorial platforms — large payment and banking teams (Harbour Bank, KiwiFirst Bank) lean on Hexawise or in-house generation scripts to manage hundreds of parameters across card types, payment channels, and currency combinations, where spreadsheet-based pairwise becomes unmanageable beyond about 10 parameters.
- Cross-team coordination is the hardest part — when 10+ squads share a microservices platform, each squad's pairwise scope must be agreed at a test-architecture level or squads duplicate effort on shared parameters and miss cross-service interaction pairs entirely; at TeleNZ scale, a missing combination in a billing–provisioning interaction has caused customer-facing outages that took days to diagnose.
◆ What I would do
Professional judgment — when to reach for pairwise testing, when to skip it, and what to watch for.
The bottom line: Pairwise testing is a modelling discipline first and a counting exercise second. The value is in being forced to identify which parameters genuinely share a code path — that question alone surfaces assumptions teams have been carrying silently. The generated test set is almost a by-product.
6 Best Practices
- ✓ Model constraints in your PICT file, not in post-processing. If Afterpay is unavailable for Click & Collect, add a constraint row to the PICT input file (
IF Payment = "Afterpay" THEN Delivery <> "Click & Collect"). Manually deleting invalid rows after generation is error-prone and easy to forget in future runs. - ✓ Version-control your PICT input files. Treat them as test assets. When a new payment method or delivery option is added, updating the .txt file and regenerating the set takes minutes. Losing the original file means someone rebuilds it by hand, which is how pairing gaps reappear.
- ✓ Audit the generated set before using it. PICT and AllPairs are reliable, but a quick spot-check matrix (a spreadsheet where rows are test cases and you tick pairs as you find them) builds confidence and catches any tool-generated anomalies. This also helps you explain coverage to non-testers.
- ✓ Add targeted tests beyond the generated set for business-critical pairs. Pairwise is a floor, not a ceiling. If your product owner says “Afterpay + rural delivery is our highest-risk combination,” verify that pair is in the generated set, and if it isn’t, add it manually rather than trust to luck.
- ✓ Use pairwise as the starting point for exploratory charters, not as a replacement for them. Pairwise gives you structured coverage of known inputs. Follow up with exploratory sessions to probe edge cases, unusual values, and the combinations that feel wrong even if the algorithm doesn’t single them out.
- ✓ Document why each parameter was included. In the test design notes, write one line per parameter explaining why it can interact with others. This forces you to think clearly about whether a parameter is genuinely independent, and it makes the rationale reviewable during a test review or audit.
- ✓ For compatibility matrices, pin your OS/browser/device versions explicitly. “Chrome” is not a value — “Chrome 126 on Windows 11” is. Vague parameters produce vague coverage. Lock versions when you generate the set, and refresh the set when major versions change.
- ✓ Reuse the pairwise set across test phases when appropriate. A pairwise set built during system testing can be re-run (with minor updates) as a regression suite. Pairwise is not throw-away work — it is a repeatable, compact interaction-coverage baseline you can schedule for every release.
- ✓ Know when to escalate to 3-way coverage. If a defect escapes pairwise and the post-mortem reveals a three-parameter interaction, add a note to the test design rationale and consider re-running with PICT’s
/o:3flag for that feature area going forward.
7 Common Misconceptions
❌ Myth: A test set is "pairwise" if it covers a lot of combinations — you just need enough rows.
Reality: Pairwise is a precise guarantee, not a description of quantity. A set of 50 randomly chosen combinations may still miss specific pairs, while a well-generated set of 9 cases can cover all pairs for a 3×3×3 parameter space. The only way to know a set is truly pairwise is to verify it against a full pair-coverage matrix, or use a verified generator like PICT. "We ran a bunch of combinations" is not pairwise testing.
❌ Myth: Pairwise testing replaces equivalence partitioning and boundary value analysis.
Reality: These techniques are complementary, not interchangeable. Equivalence partitioning defines the value classes for each individual parameter; boundary value analysis tests the edges of those classes. Pairwise testing then combines those representative values across parameters to catch interaction defects. A complete test strategy uses all three: EP/BVA to choose sensible values per parameter, then pairwise to build the combination set efficiently.
❌ Myth: Pairwise testing only makes sense for large parameter spaces — it's overkill for 3 or 4 parameters.
Reality: The value of pairwise testing scales with the number of values per parameter, not just the number of parameters. Three parameters with 5 values each gives 125 full combinations; pairwise reduces that to roughly 25. Even for modest parameter spaces, pairwise provides a structured, defensible rationale for your test selection — far better than "we tested the combinations we thought of." At four or more parameters it becomes indispensable.
8 Now You Try
Three graded exercises — spot, fix, then build. Write your answer, run it for AI feedback, then compare to the model answer.
A Foodstuffs click-and-collect checkout has three settings: store brand (New World, PAK'nSAVE, Four Square), collection slot (next hour, same day, next day), and payment (credit card, account, gift card). State the number of cases for full combinatorial coverage, then explain in your own words how many a pairwise set needs and why it is so much smaller.
Show model answer
Parameters (3, each with 3 values): - Store brand: New World, PAK'nSAVE, Four Square - Collection slot: next hour, same day, next day - Payment: credit card, account, gift card Full combinatorial: 3 x 3 x 3 = 27 test cases. Pairwise: about 9 cases. With three 3-value parameters, every pair of values can be covered in 9 well-chosen rows (the same shape as a 3x3 Latin-square layout). Why it is so much smaller: pairwise only guarantees that every PAIR of values appears together once, not every full triple. Each test case covers three pairs at once (brand+slot, brand+payment, slot+payment), so a handful of cases mops up all the pairs. Full coverage forces every three-way triple to appear, which multiplies out fast.
A tester drafted the pairwise set below for an Revenue NZ myIR login flow: device (desktop, mobile), 2FA method (SMS code, authenticator app), browser (Chrome, Safari). It claims to be a complete pairwise set, but at least one pair is never covered. Find the missing pair(s) and add the row(s) needed to complete coverage.
1. desktop — SMS code — Chrome
2. mobile — authenticator app — Safari
3. desktop — authenticator app — Chrome
List the missing pair(s) and the row(s) to add:
Show model answer
Missing pairs in the drafted set: - mobile + SMS code (never together) - mobile + Chrome (never together) - SMS code + Safari (never together) The draft only ever puts mobile with the authenticator app and Safari, so several mobile and SMS pairs go untested. A clean completing addition: 4. mobile + SMS code + Chrome (covers mobile+SMS, mobile+Chrome, SMS+Chrome already seen but fine) 5. desktop + SMS code + Safari (covers SMS+Safari, desktop+Safari) After adding rows 4 and 5, every pair across the three parameters appears at least once. The lesson: a set is not "pairwise" just because it has a few rows — you must audit it against the full list of required pairs.
An CityTransit AT HOP top-up kiosk has three parameters: card type (adult, child, tertiary), top-up amount ($5, $20, $50), and payment (EFTPOS, credit card, cash). Design a pairwise test set that covers every pair at least once, and note which tool you would use rather than hand-building it for a larger set.
Show model answer
Three 3-value parameters cover all pairs in 9 cases. One valid set: 1. adult — $5 — EFTPOS 2. adult — $20 — credit card 3. adult — $50 — cash 4. child — $5 — credit card 5. child — $20 — cash 6. child — $50 — EFTPOS 7. tertiary — $5 — cash 8. tertiary — $20 — EFTPOS 9. tertiary — $50 — credit card Every card-type+amount pair, card-type+payment pair, and amount+payment pair appears at least once. (A solver can sometimes squeeze 3x3x3 below 9, but 9 is the safe, easy-to-build answer.) Tool for larger sets: PICT (Microsoft's free CLI) or AllPairs. Never hand-build beyond 3-4 parameters — humans miss pairs, exactly the mistake in Exercise 2.
Why teams fail here
- Treating pairwise as a quantity shortcut rather than a coverage guarantee — picking “about 20 tests” without verifying every pair is actually present in the set
- Modelling UI fields as parameters instead of the logic variables those fields control — the generated set looks rigorous but the real interactions are never exercised
- Skipping constraint modelling in PICT — generating test cases for combinations the system’s own business rules make impossible, then wasting hours chasing “failures” that are invalid states
- Retiring the PICT input file instead of versioning it — when a new payment method or browser is added, nobody knows which file to update, so the set gets rebuilt by hand and pairing gaps re-emerge
Key takeaway
Pairwise testing is not about running fewer tests — it is about running the right tests: the minimal set that guarantees every two-way interaction has been exercised at least once, which is where 90% of combination defects live.
How this has changed
The field moved. Here is how Pairwise Testing evolved from its origins to current practice.
Charles Yin and Mandl independently discover that most bugs triggered by parameter combinations are exposed by testing all pairs of parameter values (as opposed to all combinations). This observation grounds pairwise testing theoretically.
Practical application grows as tools emerge. All-pairs algorithms (AETG, IPO) are published. The insight: N parameters with K values each require K^N exhaustive test cases but only O(K^2 log N) cases to cover all pairs.
James Bach and Jon Hagar promote pairwise testing at STAR conferences. PICT (Pairwise Independent Combinatorial Testing) tool released by Microsoft — free, command-line, widely adopted. Pairwise testing becomes accessible to testers without statistical backgrounds.
Research extends the theory to t-way combinatorial testing (t=2 for pairwise, t=3 for triple-wise). NIST publishes empirical data showing most bugs are triggered by 2-3 factor combinations — validating the pairwise assumption for typical software.
Cloud-based combinatorial test design tools generate pairwise sets automatically from parameter lists. AI test generation tools apply pairwise principles without users needing to know the theory. The challenge is input: defining meaningful parameters and their valid values still requires human domain knowledge.
Interview Questions
What NZ hiring managers ask about Pairwise Testing — and what strong answers look like.
A form has five dropdowns, each with four options. Why is exhaustive combinatorial testing impractical here, and how does pairwise testing help?
Strong answer: Exhaustive combinatorial requires 4^5 = 1024 test cases. Most bugs are triggered by a specific combination of two parameters (not all five) — research consistently shows that pairwise coverage catches 90%+ of parameter-combination bugs. Pairwise testing requires only around 16 test cases to cover all pairs, a 98% reduction. I use a tool like PICT (free, Microsoft) or AllPairs to generate the pairs. I supplement with domain knowledge: known-risky combinations get explicit test cases regardless of whether the pairwise algorithm selected them.
Junior/Mid
How do you handle constraints in pairwise testing — for example, where choosing one option invalidates certain combinations of other options?
Strong answer: I define constraints in the pairwise tool's configuration. In PICT, a constraint like IF [PaymentMethod] = "BankTransfer" THEN [CardNumber] = "N/A" prevents the tool from generating invalid combinations. This produces a smaller, valid test set. If the tool does not support constraints, I generate unconstrained pairs and filter out invalid combinations manually. For complex constraint networks, cause-effect graphing is a better fit — it models constraints explicitly before generating the test set, rather than filtering afterwards.
Mid/Senior
Self-Check
Click each question to reveal the answer.
Q1: What exactly does a pairwise test set guarantee — and what does it not?
It guarantees that every pair of values from any two parameters appears together in at least one test case. It does not guarantee that every full combination (every three-way or higher triple) is tested. It targets the interaction bugs that involve two settings, which research shows are the large majority.
Q2: For three parameters with 4 values each, how many cases does full combinatorial coverage need, and roughly how many does pairwise need?
Full combinatorial is 4 x 4 x 4 = 64 cases. Pairwise covers all pairs in roughly 16–20 cases — well under a third — while still catching the two-way interaction defects.
Q3: Why should you not build pairwise tables by hand for more than three or four parameters?
The pairing algorithm is fiddly and people quietly miss pairs — producing a set that looks complete but leaves gaps (exactly the failure in the fix exercise). Use a generator such as PICT or AllPairs: feed in the parameters and values, and it outputs a minimal, verified set in seconds.
Q4: Roughly what share of defects do single-parameter, pairwise, and three-way coverage each catch, per NIST research?
About 70% of defects come from a single parameter, around 90% are covered once you test all two-way (pairwise) interactions, and about 98% by three-way coverage. Pairwise gets you to roughly 90% at a fraction of the cost of full combinatorial testing.
Q5: When is pairwise not enough, and what should you do instead?
For safety-critical systems (medical, aviation) or where you already know a specific three-way interaction is risky, pairwise's two-way guarantee is insufficient — step up to three-way (or higher) combinatorial coverage, or add targeted tests for the known risky combination. Pairwise is a strong baseline, not a ceiling.
Q6: Your team is testing the Benefits NZ (Work and Income) Flexi-wage application form, which has four independent fields: employment type (full-time, part-time, casual), region (Auckland, Wellington, Christchurch, Dunedin), subsidy tier (tier 1, tier 2, tier 3), and application channel (online, in-person, phone). Full combinatorial gives 4 x 4 x 3 x 3 = 144 test cases, well beyond your sprint capacity. How would you apply pairwise here, and what trade-off are you accepting?
Feed the four parameters into PICT or AllPairs to generate a pairwise set — typically around 16–20 cases for this parameter space. The set guarantees every pair of values across any two parameters is covered at least once, catching the interaction bugs (e.g. casual employment in Dunedin with tier 3 subsidy) that slip through single-parameter testing. The trade-off you accept is that any defect requiring a specific three-way or four-way combination to trigger will not be caught unless you add targeted tests on top. For a government entitlements form at Benefits NZ, combine pairwise with risk-based analysis: identify the highest-risk combinations (e.g. part-time + Auckland + tier 1, because of volume) and verify those are present in the generated set before signing off.
Q7: What is the key difference between pairwise testing and a decision table, and when would you choose one over the other?
A decision table maps every combination of conditions to a specific expected output — it is complete and exact, but it grows exponentially (2^n columns for n binary conditions). Pairwise testing does not model expected outputs per combination; it selects a minimal set of test cases that covers all two-way input interactions, relying on general pass/fail observation rather than a pre-specified per-combination outcome. Choose a decision table when you have a small set of binary rules with well-defined business logic per combination (e.g. Revenue NZ tax-code eligibility: resident yes/no, secondary income yes/no). Choose pairwise when you have many multi-valued parameters with no clean per-combination specification and your goal is interaction coverage at scale — for example, a KiwiSaver contribution calculator with five configurable inputs each having four options.
Q8: A developer on your team says: "We've already run unit tests on each input field individually, so pairwise testing will just find the same bugs all over again — it's redundant." What is wrong with this reasoning and how do you respond?
Unit tests on individual fields catch single-parameter bugs (an invalid value accepted, a null pointer on empty input). They do not test how two fields behave when they interact in the same code path. Pairwise is designed specifically for interaction bugs — defects that only surface when a particular combination of values flows through shared logic together. The classic NZ example: each of rural delivery, credit card payment, and the surcharge engine may pass unit tests in isolation, yet the rural + credit card combination silently drops the surcharge. No amount of per-field unit testing will catch that. Respond by explaining the NIST research: roughly 20% of defects (the ones between 70% single-parameter coverage and 90% pairwise coverage) are invisible until you test pairs together. Pairwise is complementary to unit testing, not a duplicate of it.
Q9: You are reducing a 400-case regression suite for the TransitNZ online licence renewal portal, which covers browser (Chrome, Firefox, Safari, Edge), OS (Windows, macOS, iOS, Android), licence type (car, motorcycle, heavy vehicle), and RealMe login (yes, no). Your manager asks you to get the suite under 30 cases. Is pairwise the right tool, and what steps would you take?
Yes, pairwise is well suited here: four parameters with 4, 4, 3, and 2 values respectively give 4 x 4 x 3 x 2 = 96 full combinations — the reduction goal is realistic with pairwise. Steps: (1) Model the parameters in a PICT input file with explicit values; add a constraint if any combination is known to be invalid (e.g. iOS does not support heavy vehicle renewals through the portal). (2) Run PICT to generate the minimal pairwise set — expect roughly 16–20 cases. (3) Audit the output to confirm high-risk pairs (e.g. RealMe login = no + heavy vehicle, which may involve a separate credential flow) are present; add targeted cases if not. (4) Document the rationale so the PM can see that two-way interaction coverage is maintained. The result sits comfortably under 30 cases while preserving the cross-browser, cross-OS, and licence-type interaction coverage that the full suite was designed to provide.
Try It — Spot the missing pair
A NZ ferry booking system has three parameters. Someone has drafted a pairwise test set of 7 cases, but two pairs are not covered. Identify which pairs are missing.
- Payment: Credit card (CC), POLi (PO), Afterpay (AP)
- Passenger type: Adult (AD), Child (CH), Senior (SR)
- Route: Interislander (IS), Bluebridge (BB)
| Test # | Payment | Passenger | Route |
|---|---|---|---|
| 1 | CC | AD | IS |
| 2 | CC | CH | BB |
| 3 | PO | AD | BB |
| 4 | PO | CH | IS |
| 5 | AP | AD | BB |
| 6 | AP | CH | IS |
| 7 | CC | SR | IS |
Which two pairs are not covered by this test set?
Missing pairs
The two uncovered pairs are PO + SR (POLi with a Senior passenger) and AP + SR (Afterpay with a Senior passenger).
Tests 1–7 include Senior only in test #7 (CC + SR + IS). That covers CC+SR and SR+IS, but neither POLi+Senior nor Afterpay+Senior appears anywhere. Add two more tests:
| Test # | Payment | Passenger | Route | Pairs added |
|---|---|---|---|---|
| 8 | PO | SR | BB | PO+SR, SR+BB |
| 9 | AP | SR | BB | AP+SR (BB+SR already from #8) |
The key skill is auditing a pairwise set against a coverage matrix. With 3 parameters of 3/3/2 values, you need: CC×AD, CC×CH, CC×SR, PO×AD, PO×CH, PO×SR, AP×AD, AP×CH, AP×SR (9 payment+passenger pairs) plus IS/BB crossed with each payment and passenger. Any Senior row missing a payment covers a gap.
Senior engineer insight
Pairwise testing clicked for me when I stopped thinking about it as a coverage tool and started treating it as a modelling tool. The moment you sit down to list your parameters and values, you are forced to answer: which inputs actually share a code path? That question alone surfaces assumptions the team has been carrying silently. I have seen PICT runs turn up interactions the developers insisted could not exist — because nobody had drawn the dependency map before the test design meeting.
The most common mistake: teams generate a beautiful pairwise set from the wrong parameters. They list UI fields instead of the underlying logic variables those fields control. The test set is mathematically correct but practically useless because two of the “parameters” always move together in the code.
From the field
A Wellington insurance comparison platform was running a 144-case browser/OS/locale regression matrix before every fortnightly release — Chrome, Firefox, Safari, and Edge crossed with Windows, macOS, iOS, and Android, plus three locales (en-NZ, mi-NZ, Samoan) and three screen resolutions. Sprint capacity was 40 tests. A pairwise run in PICT with a single constraint (iOS only supports Safari) produced 17 cases. The team was sceptical until the first run flagged a bug: Safari on macOS with the Samoan locale was rendering the excess dropdown incorrectly — a pair nobody had tested in months. The generalisation: when multi-browser, multi-OS, or multi-locale compatibility is your concern, pairwise is not a shortcut. It is the principled method, and full combinatorial in this space is largely theatre.