Deterministic-Consistency Testing

1 The Hook

An CoverNZ team built an assistant that drafts a plain-language summary of a claim decision for a claimant. They tested it the way they test everything else: write the prompt, capture the perfect output once, save it as the expected result, assert the next run matches it exactly. Green build. They shipped the test into the nightly pipeline and went home happy.

The next morning the build was red. The morning after, green. Then red again, twice in one day, then green for three days. Nobody had touched the code. The team did what tired teams do with a test that fails for no reason: they re-ran it until it passed, and eventually they tagged it known-flaky and stopped looking. A test that is sometimes red and sometimes green for the same input teaches the team to ignore red — and an ignored red build is worse than no test at all.

What was actually happening: the model is non-deterministic. Asked to summarise the same decision, one run wrote “Your claim has been approved” and the next wrote “We have approved your claim.” Both are correct. Both say the same thing to the claimant. But character-for-character they differ, so an exact-match assertion fails one and passes the other at random. The model was behaving perfectly. The test was broken, because it asserted on the exact words of output that was never going to be identical twice.

This is the central problem of testing generative systems. The output is non-deterministic by design, so the deterministic test you reach for instinctively — assert the string equals the saved string — is guaranteed to be flaky. The skill is not making the model deterministic. It is writing assertions that hold across the variation the model will always produce.

2 The Rule

3 The Analogy

4 Why Exact-Match Goes Flaky

To test these systems you first have to understand why the instinctive assertion fails. A traditional function is deterministic: same input, same output, every time — so “assert output equals expected” is exactly right. A generative model is built to sample from many plausible continuations, so the same input legitimately produces different wording each run. Exact-match assumes a property the system does not have.

The damage is not just a failing test — it is a flaky test, one that passes and fails on the same input for no code change. Flaky tests are uniquely corrosive: the team learns that red does not mean broken, so they stop trusting red. When a real regression finally turns the build red, it hides among the flakes and ships to production. The CoverNZ team’s “known-flaky, just re-run it” tag is the exact failure mode — they had trained themselves to ignore the alarm.

The fix has two halves, and you need both. First, reduce the randomness you can control so output is as stable as the system allows. Second, assert on what is invariant — the meaning and the required facts — rather than on the wording, which will never be stable. The rest of this lesson is those two halves.

5 Temperature and Seed Control

Before you change your assertions, reduce the variation at the source. Two controls matter most.

Temperature governs how much the model samples versus picks the single most likely next token. High temperature means more varied, creative wording; low temperature (at or near zero) means the model takes the most likely path almost every time, so output is far more stable. For anything you want to test — a claim summary, a routing decision, an extraction — turn temperature down. You are not testing the model’s creativity; you are testing that it does the job consistently, and low temperature removes variation you do not need.

A seed, where the system exposes one, fixes the random starting point of sampling, so the same input and seed reproduce the same output. A seed makes a run repeatable, which is invaluable for debugging: when something fails, a fixed seed lets you reproduce the exact output instead of chasing a ghost.

The honest caveat: these reduce non-determinism, they do not eliminate it. Even at temperature zero with a fixed seed, output can still vary across model versions, hardware, or batching on the provider’s side. So temperature and seed control make testing easier — they do not let you go back to exact-match and trust it. You still need assertions that tolerate variation, because variation can still appear.

6 Semantic-Equivalence Assertions

This is the core technique that replaces exact-match. Instead of asking “does the output equal this exact string?” you ask “does the output mean the right thing?” — which is what you actually care about. “Your claim has been approved” and “We have approved your claim” are semantically equivalent and both must pass.

There are several assertion styles, from cheapest to richest, and good suites mix them:

• Required-fact / contains assertions: the output must contain the claim number, the decision (“approved”), and the review-rights line. You assert the invariant facts are present, not the wording around them. Cheap, deterministic, and catches the failures that matter most.
• Must-not-contain assertions: the output must not contain a different claimant’s name, a dollar figure that was never provided, or an internal code. Negative invariants catch dangerous variation that a “contains” check misses.
• Structural assertions: valid JSON, the required fields present, a date in the right format. The shape is invariant even when the prose is not.
• Semantic-similarity / judge assertions: for free-text meaning, compare the output to a reference answer by meaning — an embedding-similarity threshold, or a real LLM acting as a judge against your rubric. Richest, but you must validate the judge agrees with humans on a sample before trusting it.

The discipline: assert the invariants, ignore the variation. For the CoverNZ summary, the invariants are the decision, the claim number, and the review-rights notice; the variation is the exact phrasing of the opening sentence. Test the first, tolerate the second.

7 Repeat-N Stability and Tolerance Bands

A single run tells you the output was acceptable once. It does not tell you the system is consistent — and consistency is its own requirement. A claim assistant that approves a borderline case 7 times out of 10 and declines it 3 times is dangerous even if each individual answer reads fine, because the same claimant could get a different outcome on a different day.

The technique is the repeat-N stability check: run the same input N times (say 10 or 20) and measure how consistent the outputs are on the dimension that matters. For a decision, you want the decision identical every time even though the wording varies — 10/10 “approved”. For an extraction, you want the extracted value identical every time. Variation in phrasing is fine; variation in the substance is a defect.

For tasks where some numeric variation is genuinely acceptable, define a tolerance band rather than demanding an exact figure. If an assistant estimates a processing time, “10–14 days” and “about two weeks” are within tolerance; “3 days” and “six weeks” are not. The test asserts the answer falls inside the agreed band, not that it hits one exact value.

The stability check is what catches the dangerous case the single run hides: an output that is correct on the run you happened to capture, and quietly inconsistent on the substance the other times.

8 Snapshot Drift

Even with semantic assertions, teams keep a captured “known-good” output — a snapshot — as a reference. Snapshots are useful, but they carry a specific trap for generative systems: snapshot drift. Because no two runs are identical, an exact snapshot comparison fails constantly, so teams get into the habit of re-recording the snapshot whenever the test goes red — “update the snapshot, it’s just wording”.

The danger is that re-recording blindly bakes in regressions. The day the model quietly starts dropping the review-rights line, the snapshot test fails, someone re-records the snapshot to make it green, and now the broken output is the new “expected”. The safety net has been silently lowered. Snapshot drift is how a generative test slowly stops testing anything.

The fix is to snapshot the invariants, not the prose. Instead of snapshotting the whole text, snapshot the structured facts you extracted from it — decision, claim number, presence of the review-rights line, output shape. Those are stable run to run, so a change is a real signal, not noise. And never let a snapshot be re-recorded automatically on red: every update must be a reviewed, deliberate decision that the new behaviour is actually correct.

9 Common Mistakes

10 Now You Try

Three graded exercises: spot the flaky test, fix the assertion, build the consistency-test plan. Write your answer, run it for AI feedback, then compare to the model answer.

Why it is flaky: The model is non-deterministic by design, so the same input legitimately produces different wording each run. The assertion demands a character-for-character match against one captured string, so any run that phrases the summary differently fails — even when it is correct. It passes only on runs that happen to reproduce the exact saved wording, which is why it is red some nights and green others with no code change. The flakiness is in the assertion, not the model.

Are both correct? Yes. Both state the same decision (approved), the same claim number (CoverNZ-4471), and the same review right (within three months). They are semantically equivalent; only the phrasing differs. Run B is just as correct as Run A.

What should be asserted (invariants): the output CONTAINS the claim number CoverNZ-4471; it states the decision "approved"; it includes the review-rights notice (a review within three months). Optionally a must-not-contain check (no other claim number, no invented dollar figure) and a structure check.

What should be tolerated (variation): the exact opening phrasing, greetings like "Good news", and word order. None of that changes the meaning, so none of it should fail the test.

Why auto-retry is wrong: It hides the symptom instead of fixing the cause. The assertion still demands exact-match against output that is non-deterministic, so a "pass" just means one of five tries happened to reproduce the saved string. It also masks real regressions — if the model genuinely starts dropping the review-rights line, a retry can still stumble onto a good run and turn the build green, so the broken behaviour ships. Retrying a wrong assertion makes the alarm quieter, not the system safer.

Randomness controls: Set temperature low (near zero) so the model takes the most likely path and output is more stable; fix a seed if the provider exposes one so failures are reproducible for debugging. Caveat: these reduce, not eliminate, non-determinism (output can still vary across model versions/hardware), so you still need tolerant assertions — you cannot go back to exact-match.

New assertion strategy:
  - Required-fact / contains: output contains the claim number; states the correct decision ("approved"/"declined"); includes the review-rights notice (review within three months).
  - Must-not-contain: no other claimant's claim number or name; no dollar figure that was not provided; no internal code.
  - Semantic-equivalence: compare the summary to a reference by meaning (embedding-similarity threshold or a validated LLM-as-judge against a rubric) so correct paraphrases pass and a meaning change fails. Add a repeat-N check that the decision field is identical across 20 runs.

1. Randomness control: Run the test at low temperature (near zero) for stability and fix a seed if available so failures reproduce. Caveat: this reduces but does not remove non-determinism (model-version/hardware variation remains), so assertions must still tolerate wording differences. Where production runs warmer, test at the production temperature too.

2. Invariant assertions:
   - Contains: the correct decision (granted/declined), the specific benefit named, the reason for the decision, and the next-steps/appeal-rights line.
   - Must-not-contain: another person's name or client number, a dollar amount that was not supplied, internal-only codes.
   - Structure: the letter has the required sections (decision, reasons, next steps) and any required reference number is well-formed.

3. Repeat-N stability: Run the same case 20 times. MUST be identical every run: the decision, the benefit named, presence of the appeal-rights line. MAY vary: greeting, sentence order, exact phrasing of the explanation. A decision that flips across the 20 runs is a defect, not noise.

4. Tolerance band: If the letter estimates a processing or payment timeframe, accept any value inside an agreed band (e.g. "10–15 working days" or "about three weeks") and fail values outside it; do not demand one exact figure.

5. Snapshot strategy: Snapshot the extracted invariants (decision, benefit, appeal-line present, structure) rather than the full prose, so a change is a real signal. Never auto-re-record on red — every snapshot update is a reviewed decision confirming the new behaviour is genuinely correct, so a dropped appeal-rights line can never silently become the new expected.

11 Self-Check

Click each question to reveal the answer.

12 Interview Prep

Real questions asked in NZ QA interviews for AI testing roles. Read the model answers, then practise your own version.

Lessons from Production

What teams consistently discover after deploying this in real systems — things that don’t appear in documentation.

Compared to What?

Several techniques attempt to verify AI output quality. This one specifically targets non-determinism — the property that the same prompt produces different wording each run.

Use this technique alongside — not instead of — quality evaluation. Consistency without quality is a consistent fail.

When Not to Use This

Experience is knowing when a technique is not the right tool. Skip this one when:

At Enterprise Scale

Failure Analysis

Why the Business Cares