Test Lead Automation Answer Key

Expected pass output

  2 passed (8.1s)
  1 failed

  To open last HTML report run:
    npx playwright show-report

Two pass, one fail. The failure is deliberate — .shopping-cart-badge with a hyphen doesn't exist; the real selector is .shopping_cart_badge. That's what you're supposed to find by reading the trace.

Challenge solution — 1: a genuinely flaky test

Timing-based flake, not a typo. Two add-to-cart clicks fired in rapid succession with no locator-level wait:

test('cart badge updates after two adds in a row (flaky)', async ({ page }) => {
  await page.goto('https://www.saucedemo.com/');
  await page.getByPlaceholder('Username').fill('standard_user');
  await page.getByPlaceholder('Password').fill('secret_sauce');
  await page.getByRole('button', { name: 'Login' }).click();

  const addButtons = page.getByRole('button', { name: 'Add to cart' });
  await addButtons.nth(0).click();
  await addButtons.nth(1).click();  // no await between clicks, no intermediate assertion

  // Bare count check — no auto-retry, no waiting
  const badgeText = await page.locator('.shopping_cart_badge').textContent();
  expect(badgeText).toBe('2');
});

Run it five times:

npx playwright test --repeat-each=5

Most runs pass; occasional runs fail with badge text '1' because the second click's state update hasn't rendered before textContent() resolves.

Fix — use Playwright's auto-waiting assertion:

await expect(page.locator('.shopping_cart_badge')).toHaveText('2');

toHaveText polls for up to 5 seconds. The test becomes deterministic without adding a hard waitForTimeout — which would slow every run instead of just the slow ones.

Lead-level takeaway: flaky tests aren't a quality problem to fix once. They're a pattern to recognise. "Raw textContent() / innerText() without a retrying assertion" is a rule a linter could enforce, and large test suites often do.

Challenge solution — 2: publish the report as a portable artifact

Out of the box, playwright-report/ is almost portable — but trace data lives in separate .zip files that the HTML viewer loads lazily. Open index.html from a double-click, and the browser's file:// protocol blocks the fetch. Result: "No trace" on every failed test.

Fix — inline everything with the attachmentsBaseURL / embed-in-zip approach, via the built-in "open" flag combined with --reporter=html,... options, or simpler, the self-contained blob report:

# Record with a blob report, then merge that single blob into an HTML
# bundle designed for offline viewing.
npx playwright test --reporter=blob
npx playwright merge-reports --reporter=html blob-report

That produces a playwright-report/ folder with traces embedded as data URIs, viewable by double-clicking index.html.

Alternative for extreme portability — zip playwright-report/ along with the test-results/ folder (where raw trace zips live) and the teammate runs npx playwright show-report <path> locally. That spins up a tiny local server that correctly serves traces. This is how you ship a report without granting CI access.

Lead-level takeaway: "can a non-automation person open this and get value?" is the real test of a report. If they need you on a Zoom to navigate it, the report is broken.

Common mistakes

• Attaching traces to every run instead of retain-on-failure

  trace: 'on' writes a multi-MB zip per test. On a 300-test suite that's gigabytes per CI run. Use trace: 'retain-on-failure' — the setting shown in the practice config — so traces exist only when you need them.

• Assuming a "flaky" test is a Playwright bug

  Playwright's auto-waiting is thorough. If your test flakes, the cause is almost always in your code: a bare textContent(), a navigate-then-click race, or a network request your test doesn't wait for. Read the trace before blaming the framework.

• Retries hide the flake from everyone including you

  retries: 2 turns "1 in 10 runs fails" into "never seen a failure, ship it." That's a maintenance debt that blows up in 6 months. Use retries in CI for unblocking, but also track flaky test rate — tests that passed on retry but failed on first run. That's your signal.

• Sharing the report by pasting the URL into Slack

  CI artifact URLs expire (GitHub's default is 90 days; many teams drop it to 7). A link that worked last quarter is dead today. Send the bundle, or pin the run.

Expected pass output

→ GET /posts/1   [200 OK, 288B, 412ms]
  ✓ status is 200
  ✓ response has userId
→ GET /posts     [200 OK, 27.5KB, 298ms]
  ✓ status is 200
  ✓ returns an array of at least 100 items
→ POST /posts    [201 Created, 180B, 310ms]
  ✓ status is 201
  ✓ echoes back the payload
→ GET /users/1   [200 OK, 505B, 241ms]
  ✓ status is 200
  ✓ user has an email field

✓ 4 requests, 8 assertions, 0 failures

A reports/junit.xml file should exist after the run, listing 4 test cases — one per request, with the individual assertions as child elements. That XML is what CI tools (Jenkins, Azure DevOps, CircleCI) parse to populate their native test-results UI.

Challenge solution — 1: chain requests with extracted variables

Add two requests to the collection. First, POST that stores the new id:

{
  "name": "POST /posts (chained)",
  "request": {
    "method": "POST",
    "header": [{ "key": "Content-Type", "value": "application/json" }],
    "body": { "mode": "raw", "raw": "{\"title\":\"chained\",\"body\":\"x\",\"userId\":1}" },
    "url": { "raw": "{{baseUrl}}/posts", "host": ["{{baseUrl}}"], "path": ["posts"] }
  },
  "event": [{
    "listen": "test", "script": { "type": "text/javascript", "exec": [
      "pm.test('status is 201', function () { pm.response.to.have.status(201); });",
      "const body = pm.response.json();",
      "pm.test('response has an id', function () { pm.expect(body.id).to.be.a('number'); });",
      "pm.environment.set('postId', body.id);"
    ]}
  }]
}

Then a GET that uses the stored id:

{
  "name": "GET /posts/{{postId}} (chained)",
  "request": { "method": "GET", "url": { "raw": "{{baseUrl}}/posts/{{postId}}", "host": ["{{baseUrl}}"], "path": ["posts", "{{postId}}"] } },
  "event": [{
    "listen": "test", "script": { "type": "text/javascript", "exec": [
      "pm.test('status is 200 or 404 (placeholder API)', function () {",
      "  pm.expect([200, 404]).to.include(pm.response.code);",
      "});",
      "if (pm.response.code === 200) {",
      "  pm.test('returned id matches created', function () {",
      "    pm.expect(pm.response.json().id).to.eql(Number(pm.environment.get('postId')));",
      "  });",
      "}"
    ]}
  }]
}

Re-run Newman. JUnit XML should now list 12 test cases (was 8). Variable chaining is the single most common CI-collection pattern — "create X, then do Y to the thing you just created."

Interview talking point: extracting from a response and feeding it into the next request is what separates smoke tests (isolated requests) from integration tests (stateful flows). A lead who can explain that distinction confidently is a lead who can design a real API suite.

Challenge solution — 2: swap to reqres.in with just an environment change

Edit the environment file only:

{
  "name": "reqres-env",
  "values": [
    { "key": "baseUrl", "value": "https://reqres.in/api", "enabled": true }
  ]
}

Update a couple of requests to paths reqres understands — /users/2 and /users. Run:

newman run collection.json \
  -e reqres.postman_environment.json \
  -r cli,junitfull \
  --reporter-junitfull-export reports/junit.xml

The answer to "what had to change?"

• Environment: the baseUrl value.
• Collection: a handful of path fragments, because JSONPlaceholder exposes /posts and /users at root, whereas reqres uses /users under /api. Structure didn't change.
• Newman command: point at the new environment file.
• Assertions: some, because reqres wraps responses in { data: {...} } where JSONPlaceholder returns the object directly.

The business argument this unlocks: "one collection runs against dev, staging, UAT, and prod — all we change is the environment file." That's the line you use when proposing Newman in CI. Leads who can frame tools around business outcomes (cost of switching environments, audit trail, rollback safety) get buy-in faster than leads who frame them around features.

Common mistakes

• Reporter "junitfull" not found

  It's a separate package: npm install -g newman-reporter-junitfull. -r junit (built-in) works too but produces a flatter XML that some CI parsers render poorly.

• Tests pass in Postman, fail in Newman

  Usually one of three things: (a) your Postman app is reading an OS-level proxy that Newman doesn't know about — add --proxy; (b) you set an environment variable but Newman is running with --environment pointing at an older file; (c) Postman's "Current Value" for a variable is populated from manual editing and Newman reads the "Initial Value" from the exported file. Push Current Value to Initial Value before export.

• JUnit XML is empty

  The --reporter-junitfull-export path is relative to the directory Newman was invoked from. Make sure the reports/ folder exists before running — Newman won't create nested directories. Or use an absolute path.

• Every CI run hits real external APIs — rate limits, flakes, cost

  For smoke tests against jsonplaceholder this is fine. For anything hitting a paid API, use --insecure plus a mock server (WireMock, Mockoon) in CI and save the real-endpoint runs for a nightly job. Leads make this tradeoff visible.

Expected pass output

3 passed (18.4s)

The Actions tab shows a green tick next to your commit. Click into the run → there's a playwright-report artifact available for download. Artifacts retain for 7 days (per the retention-days value in the workflow).

Challenge solution — 1: shard across two jobs

Replace the single test job with a matrix:

jobs:
  test:
    name: Shard ${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
    timeout-minutes: 15
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2]
        shardTotal: [2]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report-shard-${{ matrix.shardIndex }}
          path: playwright-report/
          retention-days: 7

Push. Two jobs appear in the Actions tab, running in parallel. Two artifacts upload (playwright-report-shard-1, playwright-report-shard-2).

Wall-clock savings: on a 3-test suite, sharding is pointless — you pay setup overhead for each shard. On a 300-test suite, 2-way sharding roughly halves runtime. The break-even point is usually around 20 tests — below that, single job; above, shard.

For a truly merged report (not two separate artifacts), you'd add a third job that runs after the matrix with needs: test, downloads both artifacts, and runs npx playwright merge-reports. The practice doesn't require that; see Senior practice 01 for the pattern.

Challenge solution — 2: status badge

Create README.md at the repo root:

# Playwright CI Lab

![Playwright Tests](https://github.com/YOUR-USERNAME/playwright-ci-lab/actions/workflows/test.yml/badge.svg)

Runs Playwright smoke tests on every push and PR to main.
Report artifacts are retained for 7 days.

Replace YOUR-USERNAME/playwright-ci-lab with your actual repo path. Push. The badge renders green after the next successful run.

Why this matters: a badge is a zero-cost trust signal. Reviewers scanning a monorepo see a green (or red) badge in 200ms and know whether the tests are working. No Actions-tab click required. When presenting an automation effort to leadership, "here's the badge that's been green for 90 days" is a tighter pitch than "let me walk you through the pipeline."

Common mistakes

• Workflow doesn't appear in the Actions tab

  Three causes: (1) the file isn't at exactly .github/workflows/test.yml — any typo in the path and GitHub ignores it; (2) GitHub Actions is disabled for forks — enable in Settings → Actions → General; (3) indentation in the YAML is broken — paste into a validator before committing.

• "Process completed with exit code 1" but no obvious test failure

  Scroll up in the log. The failure often happens in npx playwright install --with-deps when apt-get hits a package mirror timeout — that exits 1 before tests even run. Re-running the workflow usually clears it.

• Uploading the report but show-report says "No data"

  You uploaded playwright-report/ but not test-results/ where trace zips live. For the standalone HTML viewer, upload both — or use the blob-merge approach from Practice 01's challenge to produce a self-contained report.

• Billing panic on the free tier

  Public repos: unlimited Actions minutes. Private repos: 2000 free minutes/month on Free plan; an average Playwright run is 2-3 minutes, so a PR-per-day project is nowhere near the limit. Sharding doubles billed minutes (2 jobs × N minutes), so on private repos weigh wall-clock vs. cost.

• Baking secrets into the workflow YAML

  Anything in test.yml is public (even on private repos, if you ever open-source). Use secrets.MY_KEY and configure in Settings → Secrets and variables → Actions. Leaked API keys from test workflows are a surprisingly common incident in real NZ teams.