20 min read · 9 self-checks · Updated June 2026

Testing Technique · Senior

Property-Based Testing

Q: What is “shrinking” in property-based testing and why is it valuable?

When the framework finds a failing input, shrinking is the process of automatically simplifying that input while the test keeps failing. The goal is the minimal reproducer — the simplest input that exposes the bug. Without shrinking, you might get a failing case like $47,283.91 and have to manually hunt for which aspect causes the failure. With shrinking, the framework hands you $0.001 and says: “this is the core of the problem.” Minimal reproducers are far easier to debug and turn into regression test cases.

Q: A property test fails on input 47,283. After shrinking, the minimal failing case is 3. What does this tell you?

The bug is triggered by any input that reaches 3 or produces 3 through the test’s logic — not by something specific to large numbers. The shrinking eliminated all the noise of the larger value. You now know: the bug appears at a small, clean input, which makes it much easier to trace through your code and find the off-by-one, divide-by-zero, or boundary condition that causes it. Start debugging at 3, not 47,283.

Q: Name three types of functions where property-based testing provides the most value.

Mathematical calculations (tax, interest, currency conversion) — properties like “GST is always 15%” cover the entire domain. Serialisation/deserialisation round-trips — the property “encode then decode returns the original” covers every possible input automatically. Sorting and filtering — properties like “the result is always a subset of the input” or “sorting never changes the element count” are universal truths that hold for any input, making them ideal for automated generation.

Q: Your team is adding property-based tests to the KiwiSaver contribution calculator on an Benefits NZ portal. The function has a database call to fetch the member’s employer match rate. Should you use property-based testing directly on this function? Why or why not?

No — not directly on that function. Property-based testing assumes pure, fast functions that can run 1,000 times in milliseconds. A function with a live database call would make 1,000 real database round-trips per test run, making the suite slow, fragile, and expensive. The correct approach is to extract the contribution calculation into a pure function that accepts the employer match rate as a parameter, then write property tests against that pure core. The database interaction is tested separately with a small number of integration tests against a seeded test database.

Q: What is the key difference between property-based testing and equivalence partitioning?

Equivalence partitioning is a manual technique: you divide the input domain into partitions and hand-pick one representative value per partition to test. Property-based testing automates this at scale — you define what must be true for the entire domain, and the framework generates hundreds or thousands of values across that domain automatically, then shrinks any failure to a minimal case. EP gives you controlled, readable, targeted examples; property-based testing searches the full input space for counterexamples your manual partitioning missed. They complement each other: EP documents known behaviour, property tests hunt for unknown failures.

Q: A developer on your team says: “I added a property-based test that passes 1,000 runs, so our Revenue NZ penalty calculation is definitely correct.” What is wrong with this claim and how do you respond?

The claim conflates passing 1,000 samples with provable correctness. A property test passing 1,000 runs means no counterexample was found — not that none exists. The strength of the guarantee depends entirely on two things: how well the generators cover the input domain, and how precisely the property defines the invariant. A weak property like “output is not null” will pass 1,000 runs even on broken code. The right response is to review the property itself: can you phrase it in one sentence that captures the Revenue NZ rule exactly? Is the generator constrained to valid NZD penalty amounts with correct precision? If both are solid, the test is strong evidence — but it is never a proof of correctness across an infinite domain.

Q: Give two scenarios from a NZ government digital service — such as RealMe identity verification or an CoverNZ claim portal — where property-based testing would NOT be the right choice, and explain what technique you would use instead.

First: testing the user journey for a RealMe login on an CoverNZ claim form — this is a UI interaction with external identity provider redirects. You cannot run 1,000 iterations against a live auth flow; use example-based end-to-end tests covering the happy path and key failure modes instead. Second: validating that a specific CoverNZ claim decision letter contains the correct claimant name, injury date, and entitlement amount — this is output that depends on a specific combination of input data, not a universal invariant. Use example-based tests with representative claim fixtures. Property-based testing is for invariants; output-shape correctness for specific business scenarios is better covered by targeted example tests or decision tables.

Instead of writing 10 test cases that you picked, let the framework generate 1,000 random test cases automatically — then shrink any failure to the simplest input that reproduces it. Property-based testing finds bugs that hand-picked examples never would.

Senior Senior SDET CTFL v4.0 — 4.2.2 · CTAL-TA v3.1.2 — 3.1.3

1 The Hook

An Auckland fintech has a GST calculation function. A developer writes 5 unit tests: $100, $50, $0, $99.99, $1000. All pass. Shipped.

A QA engineer adds property-based testing. The framework generates 1,000 random inputs. At $0.001 — one-tenth of a cent — the floating point arithmetic produces $0.00015 NZD of GST, which rounds down to $0.00. The system is charging $0.00 GST on a taxable supply.

Never visible with hand-picked examples. Found immediately with property-based testing. Revenue NZ doesn’t accept “our test inputs didn’t cover sub-cent amounts” as an audit response.

💬

Senior Engineer Insight

The most dangerous property-based test I ever reviewed passed 10,000 runs and caught nothing — because the property was "output is not null." The team shipped the calculator to an Benefits NZ portal with a rounding defect that only triggered when the contribution rate was a repeating decimal. The lesson I took from that: the value of property-based testing is 90% in the quality of the property, 10% in the run count. Before you write a single line of fast-check, write your invariant in plain English and ask: would this assertion pass on obviously broken code? If the answer is yes, you don't have a property — you have a smoke test dressed up as one. Weak generators compound the problem. I've seen teams cap fc.float at 1,000 and never realise their KiwiSaver calculator had never been exercised above that threshold in production.

Senior engineer insight

The shift that changed how I approach property-based testing was realising the test is only as strong as the invariant, not the run count. After watching a team ship a KiwiSaver rounding defect despite 10,000 passing property runs — because their property was literally “output is not null” — I now make every engineer write the invariant in a plain-English comment before touching fast-check. If you cannot state it in one sentence without referencing the implementation, you do not understand what you are testing yet.

Most common mistake: writing a property that mirrors the implementation rather than expressing an independent truth about it — which means you are testing the code against itself and every bug becomes invisible.

From the field

A Wellington payments team was building a multi-currency settlement engine for a NZ fintech operating across NZD, AUD, and USD. They had 47 example-based unit tests and had never had a production incident in six months. A senior engineer added Hypothesis tests over the weekend as a spike — within two hours the framework had found that converting NZD→AUD→NZD returned a value 0.000001 off the original due to floating-point composition, and that the discrepancy compounded across batch settlements of 500+ transactions. In a month with 40,000 settlements that was roughly $800 of unaccounted variance per settlement cycle. Revenue NZ would not have liked that. The fix was a two-line change to use Decimal throughout; the property test became a permanent regression guard. The lesson: example tests tell you your happy paths work; property tests tell you whether your mathematical foundations hold across the full domain.

2 The Rule

A property is a truth that should hold for ALL inputs in a domain, not just the ones you thought of. Test properties, not examples, and let the framework find your blind spots.

3 The Analogy

Analogy

A usability study versus asking 5 friends.

Example-based testing is asking 5 friends to try your app and checking if they can do it. Property-based testing is running a usability study with 1,000 random participants from every background, then asking the researcher to find you the single person whose experience best illustrates the core failure.

The researcher is the shrinking algorithm. When a failure is found at input $47,283.91, shrinking reduces it to the simplest case that still fails — often something like $0.001 or $100.00. The study finds the problem; the researcher hands you the clearest example of it.

4 Watch Me Do It

Property-based testing for a NZ GST calculation function using fast-check (JavaScript).

import fc from 'fast-check';

// The property: GST must always be exactly 15% of the pre-tax amount,
// total must equal pre-tax + GST, and GST must never be negative.

describe('GST Calculator — property-based tests', () => {
  it('GST is always 15% of pre-tax for any valid NZD amount', () => {
    fc.assert(
      fc.property(
        // Generate: positive numbers with at most 2 decimal places (NZD)
        fc.float({ min: 0.01, max: 1_000_000, noNaN: true })
          .map(n => Math.round(n * 100) / 100),
        (preTaxAmount) => {
          const { gst, total } = calculateGST(preTaxAmount);

          // Property 1: GST must be 15%
          const expectedGST = Math.round(preTaxAmount * 0.15 * 100) / 100;
          if (Math.abs(gst - expectedGST) > 0.005) return false;

          // Property 2: total = pretax + gst
          if (Math.abs(total - (preTaxAmount + gst)) > 0.005) return false;

          // Property 3: GST must never be negative
          if (gst < 0) return false;

          return true;
        }
      ),
      { numRuns: 1000, seed: 42 } // reproducible with seed
    );
  });

  it('Revenue NZ number validator never throws — any 8–9 digit input returns a boolean', () => {
    fc.assert(
      fc.property(
        fc.integer({ min: 10_000_000, max: 999_999_999 }),
        (num) => {
          const result = validateIRDNumber(String(num));
          // Property: result must be boolean, never throw
          return typeof result === 'boolean';
        }
      )
    );
  });
});

The same properties in Hypothesis for Python teams:

from hypothesis import given, strategies as st

@given(st.floats(min_value=0.01, max_value=1_000_000, allow_nan=False))
def test_gst_is_always_15_percent(pre_tax):
    pre_tax = round(pre_tax, 2)  # NZD precision
    gst, total = calculate_gst(pre_tax)
    assert abs(gst - round(pre_tax * 0.15, 2)) < 0.005
    assert abs(total - (pre_tax + gst)) < 0.005
    assert gst >= 0

Shrinking in action: when fast-check finds a failure at input $47,283.91, it automatically tries simpler inputs: smaller numbers, fewer decimal places, round numbers. If the real bug is “any amount with a sub-cent component”, shrinking will find $0.001 as the minimal reproducer. You get a failing test case you can understand and debug — not a random large number that’s hard to reason about. The seed parameter makes that shrunk case reproducible across runs.

5 When to Use It

Property-based testing pays off most on:

Mathematical calculations — GST, KiwiSaver interest, currency conversion, Revenue NZ penalty calculations. Any formula with a domain has properties.
Serialisation/deserialisation round-trips — encoding then decoding any valid input should return the original value. This property covers every possible input automatically.
Validation logic — any input in the valid range should produce a valid output; any input outside should be rejected. Let the framework stress-test the boundary.
Sorting and filtering — the result should always be a subset of the input; sorting should never change the element count; filtering on A then B should equal filtering on B then A.
State machines — any valid sequence of operations should leave the system in a consistent state. Random event sequences find race conditions and ordering bugs.

Don’t use property-based testing for UI interactions, network calls, or anything that can’t run 1,000 times in under a second. Keep it to pure functions and fast in-memory logic.

6 Common Mistakes

🚫 “I used to think: property-based testing replaces example-based unit tests.”

Actually: They complement each other. Examples document the intended behaviour for specific scenarios — they’re readable, targeted, and fast to debug. Properties find the edge cases your examples missed. Use both: examples to lock in known behaviour, properties to search for unknown failures. One replaces the other only if your test suite has no readers.

🚫 “I used to think: I need to generate completely random inputs with no constraints.”

Actually: Most functions have a valid input domain. Generating inputs outside that domain (negative prices, NaN, empty strings where a number is required) tests your validation logic, not your business logic. Use fast-check’s composable generators to stay in domain: fc.float({ min: 0.01, max: 1_000_000, noNaN: true }) generates realistic NZD amounts. Constrained generation finds more meaningful bugs, faster.

🚫 “I used to think: property-based tests are slow because they run 1,000 iterations.”

Actually: 1,000 in-memory function calls typically complete in under 100ms. The framework overhead of fast-check is negligible for pure functions. The test suite feels slow when you apply property-based testing to I/O-heavy or side-effectful code — which is the wrong tool. For pure calculations and data transformations, 1,000 runs is faster than you expect.

7 Industry Reality

🏭 What you actually encounter on the job

Most teams never get past one or two properties. In practice, senior engineers and QA leads spend more time convincing teammates that property-based tests are worth the learning curve than they do writing the tests themselves. Expect resistance and plan for a demo-driven rollout, not a mandate.
Legacy codebases are riddled with side effects. Property-based testing assumes pure, fast functions. In reality, you’ll encounter methods that hit the database, log to CloudWatch, or mutate global config. Retrofitting property tests to these means wrapping everything in mocks — which often makes the test suite harder to maintain than it’s worth.
Flaky seeds are a real problem. Unfixed seeds mean a property test can pass 100 times, then fail on a CI run when a scheduler change alters timing. Senior testers always pin a seed on any test that feeds into a merge gate, and document why in the test file.
Revenue NZ and financial regulators care about determinism, not coverage metrics. NZ fintechs undergoing Revenue NZ or FMA technical audits are sometimes asked to demonstrate that a calculation is provably correct across its full input range. Property tests — especially with a printed run transcript and fixed seed — are far more compelling evidence than “we tested five examples”.
Time pressure kills property design. Writing good properties requires understanding the invariants of your domain. Under sprint pressure, testers write weak properties like “output is not null” — which pass on everything including broken code. Reserve property-based testing for code that is genuinely high-stakes and stable enough to define invariants for.

8 When to Use It — and When Not To

⚡ Decision guide

✓ Use it when

The function is pure — same inputs always produce same outputs, no side effects, no I/O
You can articulate a mathematical invariant that must hold for all valid inputs (e.g. “GST is always 15%”, “round-trip encode/decode returns original”)
The input domain is large or continuous — NZD amounts, Revenue NZ numbers, dates, Unicode strings — where hand-picked examples inevitably miss edge cases
The function is high-stakes — financial calculations, compliance logic, cryptographic operations, data serialisation that crosses trust boundaries
You have a reference implementation to compare against — a simple slow version vs. an optimised fast version should produce identical outputs for any input

✗ Skip it when

The code is I/O-heavy — database calls, HTTP requests, file reads — where 1,000 iterations is impractical or expensive
The behaviour is inherently example-specific — UI interactions, user journeys, specific API contract shapes. Example-based tests are clearer here.
You can’t define a property without essentially reimplementing the function. If your “property” is just re-running the calculation a different way, you’re testing the property, not the code.
The team is unfamiliar with the framework and the sprint timeline is tight. A poorly-written property test gives false confidence; it’s worse than no property test.
The function has intentionally complex branching driven by business rules that only make sense in specific combinations — use decision tables instead.

Context guide

How the right level of property-based testing effort changes based on project context.

Context	Priority	Why
Revenue NZ tax calculation engine (e.g. GST, PAYE, KiwiSaver levy logic)	Essential	Financial calculations over a continuous NZD domain have mathematical invariants (GST is always 15%, totals must add up) that hold for all valid inputs. Manual example tests miss sub-cent and repeating-decimal edge cases that Revenue NZ auditors may find.
CoverNZ claims processing — serialisation of claim payloads to/from JSON	Essential	Encode-then-decode round-trip properties cover every possible claim shape automatically. A single property replaces dozens of example tests and will catch codec bugs before a claimant's data is silently corrupted.
Spark or Harbour Bank billing — sorting, filtering, and aggregation of transaction records	High	Ordering invariants (sorting never changes the count, filtering on A then B equals filtering on B then A) are well-defined and fast to execute. Random transaction volumes expose ordering edge cases hand-picked tests routinely miss.
Benefits NZ benefit eligibility — complex multi-condition branching driven by policy rules	Medium	Properties exist (e.g. a valid applicant always receives a decision) but the logic is policy-driven with many special cases. Property tests complement decision tables; they don't replace them. Write properties for the numerical components (payment amount arithmetic) rather than the eligibility branching.
Pacific Air check-in UI — user journey from seat selection to boarding pass	Low	UI interactions with redirects, sessions, and external APIs cannot run 1,000 times per second. There is no mathematical invariant to assert. Use example-based end-to-end tests with representative passenger personas instead.
LandNZ cadastral data API — read-only geographic parcel lookups	Low	Responses depend on specific geographic data, not universal invariants. Contract testing against a recorded response is more appropriate than random coordinate generation. Reserve property tests for any coordinate-transformation or projection functions used internally.

Trade-offs

What you gain and what you give up when you choose property-based testing.

Advantage	Disadvantage	Use instead when…
Finds edge cases across the entire input domain — sub-cent rounding errors, zero boundaries, large-number overflow — that hand-picked examples miss by definition.	Requires you to define a correct invariant before you code the test. If you can't articulate the property in one sentence, you can't write the test — and weak properties give false confidence.	The behaviour you want to verify is tied to a specific scenario rather than a universal truth — use example-based unit tests with representative fixtures.
Automatically shrinks failures to the minimal reproducing input, turning a random 47,283-digit number into a clean, debuggable case like 3 or 0.001. Saves hours of manual bisection.	Only works well on pure, fast, side-effect-free functions. Applying it to functions with database calls or HTTP requests makes the suite slow, fragile, and expensive to run.	The function has unavoidable I/O — use integration tests against a seeded test database, or extract the pure calculation core and property-test that layer only.
One concise property statement replaces dozens of manually maintained example tests, reducing test-suite maintenance burden as the domain expands (e.g. adding new NZD transaction types).	Steeper learning curve than writing example-based tests. Teams new to thinking in invariants often write properties that mirror the implementation — which tests the code against itself and hides every bug.	The team is on a tight sprint deadline and unfamiliar with the framework — a poorly-written property gives less confidence than a handful of well-chosen examples. Invest in a learning spike first.
Fixed-seed runs produce a deterministic, auditable transcript — useful for FMA or Revenue NZ technical reviews where you need to demonstrate calculation correctness across the full input domain.	Without a fixed seed, CI runs are non-deterministic. A test can pass 200 times then fail on a merge gate due to a different random sequence, eroding team trust in the suite.	The function has intentionally complex branching driven by business rules that only make sense in specific combinations — use decision tables to document the rules explicitly.

Enterprise reality

How Property-Based Testing changes at 200–300-developer scale in NZ enterprise

At small-team scale, property tests are written by whoever cares enough to add them. At enterprise scale, automation takes over: organisations like CloudBooks and Harbour Bank embed property-based test runs as a mandatory CI gate on every function that touches financial logic, with test transcripts pinned to a fixed seed and archived. The manual discipline of a single senior engineer becomes an enforced pipeline step — and the gate fails the build, not just a Slack warning.
The Privacy Act 2020 and the NZ Information Security Manual (NZISM) impose a concrete obligation at volume: any serialisation layer handling health, identity, or financial data must be demonstrably correct for all valid inputs — not just the examples a developer thought of on a Tuesday. Revenue NZ's technical audit team has requested property-test run transcripts with fixed seeds as part of GST and PAYE system reviews. "We tested representative examples" is no longer an acceptable answer for continuous-domain calculations in a regulated NZ context.
Tooling consolidates at volume. Small teams pick fast-check or Hypothesis and move on. Enterprise delivery organisations — TechServNZ, IBM NZ, the larger government SI partners — standardise on a single framework per language stack and maintain a shared generator library: nzdAmount, irdNumber, nzPhoneNumber, accInjuryCode. Generators are owned by a QA platform squad and imported as a versioned internal package. Squads that write their own generators produce subtle inconsistencies — one team's nzdAmount allows NaN; another caps at $999,999 — which produce false coverage confidence that only surfaces in production or audit.
Across 10+ squads, the failure mode is coordination, not technology. Without a QA guild enforcing property standards at PR review, sprint pressure consistently produces what practitioners call "property theatre": CI dashboards showing hundreds of passing property tests, none of which would catch a rounding defect because every property reduces to "result is a number." The antidote is a guild-owned property review checklist — "state the invariant in one sentence; verify it would fail on obviously broken code" — applied at code review, not as a post-release audit. At organisations like HealthNZ running 15+ delivery squads in parallel, skipping this coordination layer has produced identical rounding bugs independently implemented across three separate services before anyone caught the pattern.

◆ What I would do

Professional judgment — when to reach for property-based testing, when to skip it, and what to watch for.

Scenario 1 — Revenue NZ GST calculation refactor

Situation

A Wellington fintech is refactoring its GST calculation engine to comply with updated Revenue NZ guidance on mixed-supply invoices. The function is pure — it takes a line-item array and returns GST breakdowns per item and in total. The team has 12 existing example-based tests, all passing. A senior developer wants to replace them all with property-based tests to "reduce maintenance".

I would do

Keep the 12 example tests and add property tests on top — don't replace. The examples document the specific Revenue NZ mixed-supply scenarios the team has already validated; removing them loses that institutional knowledge. I would add three properties: (1) total GST equals the sum of line-item GST amounts, (2) GST on any individual line is always 15% of that line's pre-tax value, (3) total invoice value equals pre-tax total plus GST total. These properties cover the mathematical invariants across the full NZD domain. I would pin the seed and bump numRuns to 10,000 before the release cut. The combination — specific examples plus broad properties — is more defensible in an Revenue NZ audit than either approach alone.

Scenario 2 — CoverNZ claims codec

Situation

An CoverNZ digital services team is building a new claims serialisation layer that encodes claim objects to a binary protocol buffer format and decodes them back. The team has no existing tests. A junior QA asks how many test cases to write for the encoder and decoder separately.

I would do

Write a single round-trip property: "for any valid CoverNZ claim object, encode then decode returns an object equal to the original." One property replaces dozens of paired encoder/decoder examples. I would build a composite Hypothesis @composite generator or fast-check fc.record() that produces structurally valid claim objects — correct Revenue NZ numbers (8–9 digits, mod-11 check digit), valid CoverNZ injury codes, NZD amounts with two decimal places, and valid NZ phone number formats. The generator's domain constraints are where the real work lives. I would then add a small set of golden-path example tests for the three most common claim types as readable documentation for future maintainers. This approach gives 1,000-run coverage of the entire claim domain with a fraction of the manual effort.

Scenario 3 — TransitNZ toll calculation spike

Situation

TransitNZ's road tolling platform is rolling out variable pricing that adjusts toll amounts based on vehicle class, distance, and time-of-day bands. A QA lead proposes using property-based testing for the entire tolling engine, including the database lookup that fetches current rate cards.

I would do

Agree with the goal but push back on the scope. The database lookup must be separated from the calculation logic before any property tests are written — this is non-negotiable. I would extract a pure calculateToll(distanceKm, vehicleClass, timeBand, rateCard) function that accepts the rate card as a parameter rather than fetching it internally. Property tests then target this pure core: (1) toll for zero distance is always zero, (2) toll for a longer distance is always greater than or equal to toll for a shorter distance in the same band, (3) toll is always non-negative. The database interaction is tested with three or four integration tests against a seeded rate-card fixture. This split keeps the property suite fast (<200ms for 1,000 runs) and the integration suite targeted and stable.

The bottom line: Property-based testing earns its complexity when you can state an invariant in one sentence that would catch real bugs on broken code. If you can't write that sentence before opening fast-check, write the example tests first — the act of picking examples often reveals the invariant you were looking for.

9 Best Practices

✓ What experienced testers do

Always pin a seed in CI. Use { numRuns: 1000, seed: 42 } in fast-check or @settings(deriving=True) in Hypothesis. Non-deterministic failures in merge gates erode team trust in the entire test suite.
Name properties in plain language first, then code them. Write “GST must always be 15% of pre-tax amount” as a comment before writing a line of fast-check. If you can’t phrase the property in one sentence, you don’t understand it well enough to test it.
Constrain generators to the valid domain. Use fc.float({ min: 0.01, max: 1_000_000, noNaN: true }) rather than fc.float(). Unconstrained generation wastes runs on inputs that trigger validation errors, not business logic failures.
Run property tests alongside, not instead of, example-based tests. Keep your golden-path unit tests. Property tests search for unknown failures; example tests document known behaviour. Both serve different readers.
Add a beforeAll / @example decorator with the shrunk failing case as a regression. When a property test finds a bug, add the minimal reproducer as a standalone example-based test. That way it stays fast and readable in future runs even if you remove the property test.
Increase numRuns for high-stakes code before release. 1,000 runs is the default. For NZ financial calculations going into production, bump to 10,000 in a pre-release run. The extra cost (usually under 2 seconds) is trivial; the coverage gain is real.
Write a custom arbitrary for your domain objects. If you have a NZDAmount type or an IRDNumber value object, write one fc.record() or Hypothesis composite generator for it and reuse it across all property tests. Keeps generators consistent and domain-correct.
Log the failing input before shrinking completes. Add a console.log or Hypothesis note() in the test body. If shrinking produces an unexpected minimal case, the original failure gives you context the minimal case doesn’t.
Treat a shrunk failure as a specification gap, not just a bug. When the framework finds a case your spec didn’t anticipate — like sub-cent GST amounts — update the requirement document. The property test found a gap in the spec, not just the code.
Review property test output with the developer, not just the report. The shrunk minimal case is only meaningful if someone understands why it fails. Walk through it together; the conversation almost always surfaces a deeper design issue.

10 Common Misconceptions

❌ Myth: Property-based testing is only for functional programmers and Haskell enthusiasts.

Reality: Fast-check works in vanilla JavaScript and TypeScript with no functional programming background required. Hypothesis runs in standard Python. The concept — “define what should always be true, let the framework find counterexamples” — is simpler than most parametrised test setups teams already use. The barrier is conceptual (thinking in invariants, not examples), not technological. NZ teams using Jest or pytest can add property-based tests to an existing suite in an afternoon.

❌ Myth: If the property test passes 1,000 runs, the function is correct.

Reality: A property test passing 1,000 runs means no counterexample was found in 1,000 samples — not that no counterexample exists. The confidence level depends on the size of the input domain, the quality of the generator, and the cleverness of the property. A weak property (e.g. “output is not null”) that passes 1,000 runs tells you almost nothing. Strong, domain-specific invariants with well-constrained generators give genuine confidence. Always think critically about what your property actually asserts.

❌ Myth: Shrinking always gives you the “real” root cause of the bug.

Reality: Shrinking gives you the minimal input that fails the test, not necessarily the minimal input that explains the bug. If your property is poorly defined or your generators are unconstrained, the shrunk case can be an artefact of the generator’s reduction path, not a meaningful example. For instance, shrinking an unconstrained float might give you NaN as the minimal case — which fails because your code doesn’t handle NaN, not because of any business logic error. Always validate that the shrunk case makes sense in domain terms.

11 Now You Try

📋 Prompt Lab — Write Properties for an Exchange Rate Converter

A NZ exchange rate converter takes an NZD amount and a currency code ('USD', 'AUD', 'GBP', 'EUR') and returns the converted amount. Write 3 properties for this function using fast-check: (1) converting to the same currency returns the original amount, (2) converting NZD→USD then USD→NZD returns approximately the original amount (within 1%), (3) a positive NZD amount always produces a positive result in any supported currency.

Why teams fail here

Weak properties that pass on broken code — assertions like “output is not null” or “result is a number” give false confidence; they would pass even if the calculation was returning random garbage.
Unconstrained generators — using fc.float() or st.floats() without bounds floods the test with NaN, Infinity, and negative values that hit validation guards rather than the business logic you actually care about.
Applying property-based testing to I/O-heavy functions — running 1,000 iterations against functions that call a database, hit an HTTP endpoint, or write to disk makes the suite slow, expensive, and flaky; extract the pure calculation core first.
No fixed seed in CI — unfixed seeds mean a property test can pass 200 times then fail on a merge gate due to a different random sequence; always pin a seed for deterministic CI runs and document the reason in the test file.

Key takeaway

Property-based testing is not about running more tests — it is about defining what must always be true, then letting a machine find the one input in ten thousand where it is not.

How this has changed

The field moved. Here is how Property-Based Testing evolved from its origins to current practice.

1984

QuickCheck conceived by Koen Claessen and John Hughes at Chalmers University (Haskell implementation, 1999). The idea: specify properties that should hold for all inputs, let the framework generate hundreds of random inputs, and automatically shrink failures to minimal reproducing cases.

2000s

QuickCheck ported to dozens of languages. ScalaCheck, PropEr (Erlang), and RapidCheck (C++) bring property-based testing to non-Haskell teams. Used primarily by functional programming communities where property specification fits naturally.

2011

Hypothesis (Python) by David MacIver significantly improves the developer experience — better failure shrinking, stateful testing, and integration with pytest. Property-based testing begins reaching mainstream Python teams.

2015

fast-check (JavaScript), jqwik (Java), and SwiftCheck make property-based testing accessible in the most common enterprise languages. Property-based testing appears in testing certification syllabi and conference talks.

Now

AI tools can suggest testable properties from function signatures and documentation — reducing the hardest part of property-based testing (thinking of what properties to specify). LLM output testing benefits from property-based approaches: "the summary should be shorter than the input" is a property assertion that scales to AI system evaluation.

12 Self-Check

Click each question to reveal the answer.

Interview Questions

What NZ hiring managers ask about Property-Based Testing — and what strong answers look like.

What is a property in property-based testing, and how does it differ from a test case?

Strong answer: A test case checks one specific input/output pair: "given input X, expect output Y." A property specifies something that should be true for all inputs: "for any string, reversing it twice should return the original string." The property-based framework generates hundreds or thousands of inputs, including edge cases a human would not think to write, and verifies the property holds for all of them. When a property fails, the framework automatically shrinks the failing case to the minimal example. Properties are more concise than exhaustive test cases and surface bugs from unexpected input combinations.

Junior/Mid

Give an example of a good property for a function that applies a discount to a NZ GST-inclusive price.

Strong answer: Good properties: (1) Identity — a 0% discount returns the original price; (2) Range — the discounted price should always be between 0 and the original price inclusive; (3) Monotonicity — a 20% discount always produces a lower price than a 10% discount for any positive input; (4) Round-trip — applying a 10% discount and then a -10% adjustment should return approximately the original price (within rounding); (5) GST invariant — the GST component of the discounted price should be the same percentage of the new price as the original. These properties catch off-by-one errors in percentage calculation, incorrect rounding modes, and negative price edge cases that a single example-based test would not.

Mid/Senior

Q1: What is “shrinking” in property-based testing and why is it valuable?

When the framework finds a failing input, shrinking is the process of automatically simplifying that input while the test keeps failing. The goal is the minimal reproducer — the simplest input that exposes the bug. Without shrinking, you might get a failing case like $47,283.91 and have to manually hunt for which aspect causes the failure. With shrinking, the framework hands you $0.001 and says: “this is the core of the problem.” Minimal reproducers are far easier to debug and turn into regression test cases.

Q2: A property test fails on input 47,283. After shrinking, the minimal failing case is 3. What does this tell you?

The bug is triggered by any input that reaches 3 or produces 3 through the test’s logic — not by something specific to large numbers. The shrinking eliminated all the noise of the larger value. You now know: the bug appears at a small, clean input, which makes it much easier to trace through your code and find the off-by-one, divide-by-zero, or boundary condition that causes it. Start debugging at 3, not 47,283.

Q3: Name three types of functions where property-based testing provides the most value.

Mathematical calculations (tax, interest, currency conversion) — properties like “GST is always 15%” cover the entire domain. Serialisation/deserialisation round-trips — the property “encode then decode returns the original” covers every possible input automatically. Sorting and filtering — properties like “the result is always a subset of the input” or “sorting never changes the element count” are universal truths that hold for any input, making them ideal for automated generation.

Q4: Your team is adding property-based tests to the KiwiSaver contribution calculator on an Benefits NZ portal. The function has a database call to fetch the member’s employer match rate. Should you use property-based testing directly on this function? Why or why not?

No — not directly on that function. Property-based testing assumes pure, fast functions that can run 1,000 times in milliseconds. A function with a live database call would make 1,000 real database round-trips per test run, making the suite slow, fragile, and expensive. The correct approach is to extract the contribution calculation into a pure function that accepts the employer match rate as a parameter, then write property tests against that pure core. The database interaction is tested separately with a small number of integration tests against a seeded test database.

Q5: What is the key difference between property-based testing and equivalence partitioning?

Equivalence partitioning is a manual technique: you divide the input domain into partitions and hand-pick one representative value per partition to test. Property-based testing automates this at scale — you define what must be true for the entire domain, and the framework generates hundreds or thousands of values across that domain automatically, then shrinks any failure to a minimal case. EP gives you controlled, readable, targeted examples; property-based testing searches the full input space for counterexamples your manual partitioning missed. They complement each other: EP documents known behaviour, property tests hunt for unknown failures.

Q6: A developer on your team says: “I added a property-based test that passes 1,000 runs, so our Revenue NZ penalty calculation is definitely correct.” What is wrong with this claim and how do you respond?

The claim conflates passing 1,000 samples with provable correctness. A property test passing 1,000 runs means no counterexample was found — not that none exists. The strength of the guarantee depends entirely on two things: how well the generators cover the input domain, and how precisely the property defines the invariant. A weak property like “output is not null” will pass 1,000 runs even on broken code. The right response is to review the property itself: can you phrase it in one sentence that captures the Revenue NZ rule exactly? Is the generator constrained to valid NZD penalty amounts with correct precision? If both are solid, the test is strong evidence — but it is never a proof of correctness across an infinite domain.

Q7: Give two scenarios from a NZ government digital service — such as RealMe identity verification or an CoverNZ claim portal — where property-based testing would NOT be the right choice, and explain what technique you would use instead.

First: testing the user journey for a RealMe login on an CoverNZ claim form — this is a UI interaction with external identity provider redirects. You cannot run 1,000 iterations against a live auth flow; use example-based end-to-end tests covering the happy path and key failure modes instead. Second: validating that a specific CoverNZ claim decision letter contains the correct claimant name, injury date, and entitlement amount — this is output that depends on a specific combination of input data, not a universal invariant. Use example-based tests with representative claim fixtures. Property-based testing is for invariants; output-shape correctness for specific business scenarios is better covered by targeted example tests or decision tables.

13 ISTQB Mapping

CTFL v4.0 — Section 4.2.2: Equivalence partitioning

Property-based testing automates partition coverage. Rather than manually selecting one value per partition, the framework generates values across the entire input domain and verifies the property holds in all of them. It is EP taken to its logical extreme: instead of one representative, use every representative the domain contains.

CTAL-TA v3.1.2 — Section 3.1.3: Test design — automated test case generation: property specification is a form of automated test design. The property replaces the manual test case: instead of writing “input $100, expect $15 GST”, you write “for any valid NZD amount, GST must be 15%” and the framework generates the test cases. This maps to what the syllabus calls specification-based automated test generation.

14 Next Steps

Property-based testing finds bugs in pure logic. For finding bugs in your tests themselves, explore Mutation Testing. For applying the same rigour to complex data transformations involving containers and databases, look at TestContainers.

Property-Based Testing

1 The Hook

2 The Rule

3 The Analogy

4 Watch Me Do It

5 When to Use It

6 Common Mistakes

7 Industry Reality

8 When to Use It — and When Not To

✓ Use it when

✗ Skip it when

Context guide

Trade-offs

◆ What I would do

9 Best Practices

10 Common Misconceptions

11 Now You Try

How this has changed

12 Self-Check

Interview Questions

13 ISTQB Mapping

14 Next Steps

Prerequisites

Related Techniques

What to Learn Next

Also in Bootcamp

Property-Based Testing

1 The Hook

2 The Rule

3 The Analogy

4 Watch Me Do It

5 When to Use It

6 Common Mistakes

7 Industry Reality

8 When to Use It — and When Not To

✓ Use it when

✗ Skip it when

Context guide

Trade-offs

◆ What I would do

9 Best Practices

10 Common Misconceptions

11 Now You Try

How this has changed

Related techniques

12 Self-Check

Interview Questions

13 ISTQB Mapping

14 Next Steps

Prerequisites

Related Techniques

What to Learn Next

Also in Bootcamp