Test Reporting & Artefacts

1 The Hook — Why This Matters

An Auckland SaaS team runs 800 Playwright tests in CI. The pass rate is 97.2%. The Test Lead is diligent — every sprint they assemble a 40-page Allure report and email it to the Product Owner. The PO skips it every single sprint. Too much detail, no clear action.

After three sprints, a P1 bug reaches production. The automated tests had caught it. The failure was right there in the Allure report, buried on page 28. Nobody triaged it. The PO didn't know failing tests could sit unremediated if nobody reviewed the dashboard.

The tests were doing their job. The reporting wasn't. A 40-page report with no executive summary is not a communication tool — it's a data dump that creates a false sense of oversight. The PO assumed a report landing in their inbox meant someone had reviewed it.

2 The Rule — The One-Sentence Version

Three different audiences need three different formats. A developer debugging a 2am failure needs trace viewer access. A test lead needs suite-level trends. A Product Owner in a Go/No-Go meeting needs four sentences. Match the format to the audience.

3 The Analogy — Think Of It Like...

4 Watch Me Do It — Three Reports for Three Audiences

# After a test run, open the HTML report locally
npx playwright show-report

# CI uploads it as an artefact — download and run:
npx playwright show-report playwright-report/

// BAD: tells you nothing when it fails
test('test_login_01', async ({ page }) => { ... });

// GOOD: tells you exactly what scenario broke
test('KiwiSaver enrolment completes when IRD number is valid', async ({ page }) => {
  // Developer reading the failure report knows immediately:
  // the IRD number validation path broke, not the login form
});

npm install allure-playwright --save-dev

# playwright.config.ts — add the reporter
reporter: [['allure-playwright']],

# Generate and open the report
npx allure generate allure-results --clean
npx allure open

import { test } from '@playwright/test';
import { allure } from 'allure-playwright';

test('payment gateway processes Visa card correctly', async ({ page }) => {
  allure.story('PAY-1142');
  allure.severity('critical');
  allure.feature('Payment Gateway');
  // test steps...
});

Sprint 24 — Test Summary: 312 automated tests ran across 3 browsers.
308 passed (98.7%). 4 failures: 2 blocked on BUG-4421 (payment gateway
timeout — known, non-blocking for this release), 1 flaky network test
(quarantined, not counted against release quality), 1 new failure in
checkout (BUG-4438 raised, P2). No release blockers as of 09:00 Thursday.

Go recommendation: conditional on BUG-4438 resolution by EOD Thursday.
Assigned to Aaron. If not resolved, escalate to release manager by 16:00.

5 When to Use It — Audience Matrix

Report flaky tests separately from genuine failures. A 95% pass rate with 40 flaky tests is a different risk profile to 95% with zero flaky tests. The executive summary should always call out the flaky test count as a distinct number.

6 Common Mistakes — Don't Do This

7 Now You Try — Prompt Lab

Write your answer in the box below and the AI coach will review it.

8 Self-Check — Can You Actually Do This?

Click each question to reveal the answer.

9 ISTQB Mapping

10 Next Steps

You now know how to surface test results to three different audiences without losing information or burying it. Apply both skills in the practice section.