Testing Technique · Senior

Property-Based Testing

Instead of writing 10 test cases that you picked, let the framework generate 1,000 random test cases automatically — then shrink any failure to the simplest input that reproduces it. Property-based testing finds bugs that hand-picked examples never would.

Senior Senior SDET CTFL v4.0 — 4.2.2 · CTAL-TA v3.1.2 — 3.1.3

1 The Hook

An Auckland fintech has a GST calculation function. A developer writes 5 unit tests: $100, $50, $0, $99.99, $1000. All pass. Shipped.

A QA engineer adds property-based testing. The framework generates 1,000 random inputs. At $0.001 — one-tenth of a cent — the floating point arithmetic produces $0.00015 NZD of GST, which rounds down to $0.00. The system is charging $0.00 GST on a taxable supply.

Never visible with hand-picked examples. Found immediately with property-based testing. IRD doesn’t accept “our test inputs didn’t cover sub-cent amounts” as an audit response.

2 The Rule

A property is a truth that should hold for ALL inputs in a domain, not just the ones you thought of. Test properties, not examples, and let the framework find your blind spots.

3 The Analogy

Analogy

A usability study versus asking 5 friends.

Example-based testing is asking 5 friends to try your app and checking if they can do it. Property-based testing is running a usability study with 1,000 random participants from every background, then asking the researcher to find you the single person whose experience best illustrates the core failure.

The researcher is the shrinking algorithm. When a failure is found at input $47,283.91, shrinking reduces it to the simplest case that still fails — often something like $0.001 or $100.00. The study finds the problem; the researcher hands you the clearest example of it.

4 Watch Me Do It

Property-based testing for a NZ GST calculation function using fast-check (JavaScript).

import fc from 'fast-check';

// The property: GST must always be exactly 15% of the pre-tax amount,
// total must equal pre-tax + GST, and GST must never be negative.

describe('GST Calculator — property-based tests', () => {
  it('GST is always 15% of pre-tax for any valid NZD amount', () => {
    fc.assert(
      fc.property(
        // Generate: positive numbers with at most 2 decimal places (NZD)
        fc.float({ min: 0.01, max: 1_000_000, noNaN: true })
          .map(n => Math.round(n * 100) / 100),
        (preTaxAmount) => {
          const { gst, total } = calculateGST(preTaxAmount);

          // Property 1: GST must be 15%
          const expectedGST = Math.round(preTaxAmount * 0.15 * 100) / 100;
          if (Math.abs(gst - expectedGST) > 0.005) return false;

          // Property 2: total = pretax + gst
          if (Math.abs(total - (preTaxAmount + gst)) > 0.005) return false;

          // Property 3: GST must never be negative
          if (gst < 0) return false;

          return true;
        }
      ),
      { numRuns: 1000, seed: 42 } // reproducible with seed
    );
  });

  it('IRD number validator never throws — any 8–9 digit input returns a boolean', () => {
    fc.assert(
      fc.property(
        fc.integer({ min: 10_000_000, max: 999_999_999 }),
        (num) => {
          const result = validateIRDNumber(String(num));
          // Property: result must be boolean, never throw
          return typeof result === 'boolean';
        }
      )
    );
  });
});

The same properties in Hypothesis for Python teams:

from hypothesis import given, strategies as st

@given(st.floats(min_value=0.01, max_value=1_000_000, allow_nan=False))
def test_gst_is_always_15_percent(pre_tax):
    pre_tax = round(pre_tax, 2)  # NZD precision
    gst, total = calculate_gst(pre_tax)
    assert abs(gst - round(pre_tax * 0.15, 2)) < 0.005
    assert abs(total - (pre_tax + gst)) < 0.005
    assert gst >= 0

Shrinking in action: when fast-check finds a failure at input $47,283.91, it automatically tries simpler inputs: smaller numbers, fewer decimal places, round numbers. If the real bug is “any amount with a sub-cent component”, shrinking will find $0.001 as the minimal reproducer. You get a failing test case you can understand and debug — not a random large number that’s hard to reason about. The seed parameter makes that shrunk case reproducible across runs.

5 When to Use It

Property-based testing pays off most on:

Mathematical calculations — GST, KiwiSaver interest, currency conversion, IRD penalty calculations. Any formula with a domain has properties.
Serialisation/deserialisation round-trips — encoding then decoding any valid input should return the original value. This property covers every possible input automatically.
Validation logic — any input in the valid range should produce a valid output; any input outside should be rejected. Let the framework stress-test the boundary.
Sorting and filtering — the result should always be a subset of the input; sorting should never change the element count; filtering on A then B should equal filtering on B then A.
State machines — any valid sequence of operations should leave the system in a consistent state. Random event sequences find race conditions and ordering bugs.

Don’t use property-based testing for UI interactions, network calls, or anything that can’t run 1,000 times in under a second. Keep it to pure functions and fast in-memory logic.

6 Common Mistakes

🚫 “I used to think: property-based testing replaces example-based unit tests.”

Actually: They complement each other. Examples document the intended behaviour for specific scenarios — they’re readable, targeted, and fast to debug. Properties find the edge cases your examples missed. Use both: examples to lock in known behaviour, properties to search for unknown failures. One replaces the other only if your test suite has no readers.

🚫 “I used to think: I need to generate completely random inputs with no constraints.”

Actually: Most functions have a valid input domain. Generating inputs outside that domain (negative prices, NaN, empty strings where a number is required) tests your validation logic, not your business logic. Use fast-check’s composable generators to stay in domain: fc.float({ min: 0.01, max: 1_000_000, noNaN: true }) generates realistic NZD amounts. Constrained generation finds more meaningful bugs, faster.

🚫 “I used to think: property-based tests are slow because they run 1,000 iterations.”

Actually: 1,000 in-memory function calls typically complete in under 100ms. The framework overhead of fast-check is negligible for pure functions. The test suite feels slow when you apply property-based testing to I/O-heavy or side-effectful code — which is the wrong tool. For pure calculations and data transformations, 1,000 runs is faster than you expect.

7 Now You Try

📋 Prompt Lab — Write Properties for an Exchange Rate Converter

A NZ exchange rate converter takes an NZD amount and a currency code ('USD', 'AUD', 'GBP', 'EUR') and returns the converted amount. Write 3 properties for this function using fast-check: (1) converting to the same currency returns the original amount, (2) converting NZD→USD then USD→NZD returns approximately the original amount (within 1%), (3) a positive NZD amount always produces a positive result in any supported currency.

8 Self-Check

Click each question to reveal the answer.

Q1: What is “shrinking” in property-based testing and why is it valuable?

When the framework finds a failing input, shrinking is the process of automatically simplifying that input while the test keeps failing. The goal is the minimal reproducer — the simplest input that exposes the bug. Without shrinking, you might get a failing case like $47,283.91 and have to manually hunt for which aspect causes the failure. With shrinking, the framework hands you $0.001 and says: “this is the core of the problem.” Minimal reproducers are far easier to debug and turn into regression test cases.

Q2: A property test fails on input 47,283. After shrinking, the minimal failing case is 3. What does this tell you?

The bug is triggered by any input that reaches 3 or produces 3 through the test’s logic — not by something specific to large numbers. The shrinking eliminated all the noise of the larger value. You now know: the bug appears at a small, clean input, which makes it much easier to trace through your code and find the off-by-one, divide-by-zero, or boundary condition that causes it. Start debugging at 3, not 47,283.

Q3: Name three types of functions where property-based testing provides the most value.

Mathematical calculations (tax, interest, currency conversion) — properties like “GST is always 15%” cover the entire domain. Serialisation/deserialisation round-trips — the property “encode then decode returns the original” covers every possible input automatically. Sorting and filtering — properties like “the result is always a subset of the input” or “sorting never changes the element count” are universal truths that hold for any input, making them ideal for automated generation.

9 ISTQB Mapping

CTFL v4.0 — Section 4.2.2: Equivalence partitioning

Property-based testing automates partition coverage. Rather than manually selecting one value per partition, the framework generates values across the entire input domain and verifies the property holds in all of them. It is EP taken to its logical extreme: instead of one representative, use every representative the domain contains.

CTAL-TA v3.1.2 — Section 3.1.3: Test design — automated test case generation: property specification is a form of automated test design. The property replaces the manual test case: instead of writing “input $100, expect $15 GST”, you write “for any valid NZD amount, GST must be 15%” and the framework generates the test cases. This maps to what the syllabus calls specification-based automated test generation.

10 Next Steps

Property-based testing finds bugs in pure logic. For finding bugs in your tests themselves, explore Mutation Testing. For applying the same rigour to complex data transformations involving containers and databases, look at TestContainers.

Mutation Testing Boundary Value Analysis TestContainers →