Senior Automation Answer Key

Expected pass output

Running 12 tests using 4 workers

  12 passed (18.3s)

To open last HTML report run:

  npx playwright show-report

12 = 4 tests × 3 browsers. If you see 4, you dropped the three-project config. If you see 8, WebKit failed to install (common on older Windows/Linux).

Challenge solution — 1: merge sharded reports

Two shards, both emit a blob report, then a separate command merges them:

# Shard 1
PLAYWRIGHT_BLOB_REPORT_OUTPUT_DIR=blob-report-1 \
  npx playwright test --shard=1/2 --reporter=blob

# Shard 2 (in a different job / terminal)
PLAYWRIGHT_BLOB_REPORT_OUTPUT_DIR=blob-report-2 \
  npx playwright test --shard=2/2 --reporter=blob

# After both finish, merge into one HTML report
npx playwright merge-reports --reporter=html blob-report-1 blob-report-2

On Windows PowerShell the env-var syntax is different:

$env:PLAYWRIGHT_BLOB_REPORT_OUTPUT_DIR = "blob-report-1"
npx playwright test --shard=1/2 --reporter=blob

The merged playwright-report/ folder should contain all 12 tests across 3 browsers. Open playwright-report/index.html in a browser to verify.

Why blob + merge and not just two HTML reports? HTML reports can't be combined — the file structure assumes one report per run. Blob is a portable intermediate format designed for cross-job aggregation. In CI you upload blobs as artifacts from each shard job, download all of them in a final job, merge, and publish. That's the canonical pattern for sharded Playwright on GitHub Actions / Buildkite / Jenkins.

Challenge solution — 2: add mobile Safari

Append a fourth project to playwright.config.ts:

{
  name: 'mobile-safari',
  use: { ...devices['iPhone 14'] },
},

Run npx playwright test. Test count jumps from 12 to 16 (4 tests × 4 projects).

Watch for: devices['iPhone 14'] is WebKit under the hood. If WebKit didn't install (see common mistakes below), mobile Safari will also fail. npx playwright install webkit fixes both.

Common mistakes

• Only Chromium actually runs — Firefox and WebKit skipped silently

  Your projects array is correct, but the browser binaries aren't installed. Run npx playwright install (no browser name) to pull all three. On Linux CI also add --with-deps to install system libraries for WebKit.

• Passing --shard=1 instead of --shard=1/2

  Sharding needs the total — --shard=<index>/<total>. Without the total, Playwright can't compute the split. Error is explicit: "Sharding requires a value like --shard=1/3, got '1'".

• Retries set to 0 locally but high in CI — flaky tests only surface in production

  The config template uses retries: process.env.CI ? 2 : 0. This is standard but it means you never see flakiness on your laptop. Once a month, run npx playwright test --retries=0 locally on the full suite and chase any intermittent failures.

• Running all projects when you meant one

  Use --project=chromium (equals sign, no space). Space means the next token is interpreted as a file path.

• WebKit works locally but fails in GitHub Actions on Ubuntu

  Ubuntu runners need libnss3, libatk1.0-0, and friends. Always run npx playwright install --with-deps webkit in the workflow; the --with-deps flag installs the apt packages too.

Expected pass output

POST https://reqres.in/api/login          200 OK
GET  https://httpbin.org/bearer            200 OK
GET  https://httpbin.org/basic-auth/...    200 OK

9/9 tests passed

Three requests, nine assertions (3 + 3 + 3). authToken should appear in the Postman console log from Request 1 and be echoed back as body.token in Request 2's response.

Challenge solution — 1: collection-level pre-request auth

Click the collection name → Pre-request Script tab (not on a single request — the collection root). Paste:

const existing = pm.environment.get("authToken");

if (!existing) {
    console.log("No authToken in environment — logging in first");
    pm.sendRequest({
        url: "https://reqres.in/api/login",
        method: "POST",
        header: {
            "Content-Type": "application/json",
            "x-api-key": "reqres-free-v1"
        },
        body: {
            mode: "raw",
            raw: JSON.stringify({
                email: "eve.holt@reqres.in",
                password: "cityslicka"
            })
        }
    }, function (err, res) {
        if (err) {
            console.error("Pre-request login failed:", err);
            return;
        }
        const token = res.json().token;
        pm.environment.set("authToken", token);
        console.log("authToken set by pre-request hook:", token);
    });
}

Clear the environment's authToken value, then re-run the collection. The first request still fires the login (that's the explicit one), but future runs that start from a "clean slate" will self-heal.

Real-world use: on long-running CI collections, tokens expire. This hook is usually wrapped with a "is the token expired?" check using jwt_decode or the issued-at claim. For this exercise, missing-token is enough.

Challenge solution — 2: run headless with Newman

Export collection + environment from Postman (three-dot menu → Export, Collection v2.1).

npm install -g newman

newman run Bootcamp-Auth-Flow.postman_collection.json \
  -e bootcamp-local.postman_environment.json

Expected:

executed    failed
iterations        1        0
  requests        3        0
test-scripts      6        0
prerequest-scripts 1       0
assertions        9        0

Nine assertions, zero failures, no Postman app open. This is the CI endpoint — any runner that can execute an npm command can run this.

Handling the token in CI: never commit an environment file with real secrets. The authToken value should be blank in the committed environment; the pre-request hook from Challenge 1 populates it at runtime. For longer-lived tokens (OAuth refresh tokens, API keys), use --env-var: newman run collection.json --env-var "apiKey=$API_KEY".

Common mistakes

• Login returns 401 "Missing API key"

  reqres.in now requires x-api-key: reqres-free-v1. It's free, public, and unlimited — but the header is mandatory. Add it to the login request headers. The same API key works for every endpoint; don't generate your own.

• Bearer request sends Authorization: Bearer {{authToken}} literally

  You set the Authorization field using the Headers tab instead of the Authorization tab. Postman's variable resolution runs on the Authorization tab's "Token" value, but not on raw header values when you configured them as literal strings. Use the Authorization tab.

• Setting the token but reading it from the wrong scope

  pm.environment.set("authToken", ...) writes to the active environment. {{authToken}} in a URL or header is resolved through the variable lookup order: data → environment → collection → global. If no environment is selected, the write goes nowhere and the lookup falls back to collection/global and finds nothing.

• Tests appear to pass but no test results show

  You pasted the test code into the Pre-request Script tab instead of the Tests tab. Pre-request runs before the response exists — pm.response is undefined, and your pm.test calls silently do nothing. Move the script to the Tests tab.

• Token logged to console in CI — leaks into build logs

  The console.log("authToken set to:", token) line is fine locally but a leak risk in CI. Remove the log, or guard it with if (!pm.environment.get("CI")). Treat build logs as public.

Expected pass output

TestCase [CalculatorSoap TestCase] ran with status: FINISHED
  Time Taken: 312ms
  Test Step Results:
    Add 7 + 5 — PASSED

Total Request Assertions: 4
Total Failed Assertions: 0

Four assertions pass: SOAP Response valid, Schema Compliance matches the WSDL, XPath returns 12, and Not SOAP Fault.

Challenge solution — 1: data-driven across all four operations

SoapUI Open Source doesn't have the Pro DataSource, but Groovy + Properties gets you there. Right-click the test case → Add Step → Groovy Script. Paste:

import groovy.xml.XmlSlurper

def cases = [
  [op: 'Add',      a: 7,  b: 5,  expected: 12],
  [op: 'Add',      a: -1, b: 1,  expected: 0],
  [op: 'Subtract', a: 10, b: 3,  expected: 7],
  [op: 'Subtract', a: 0,  b: 5,  expected: -5],
  [op: 'Multiply', a: 4,  b: 6,  expected: 24],
  [op: 'Multiply', a: 0,  b: 99, expected: 0],
  [op: 'Divide',   a: 20, b: 4,  expected: 5],
  [op: 'Divide',   a: 7,  b: 2,  expected: 3],  // integer division
]

def endpoint = 'http://www.dneonline.com/calculator.asmx'

cases.each { c ->
  def soapAction = "http://tempuri.org/${c.op}"
  def body = """<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:tem="http://tempuri.org/">
    <soapenv:Header/>
    <soapenv:Body>
      <tem:${c.op}>
        <tem:intA>${c.a}</tem:intA>
        <tem:intB>${c.b}</tem:intB>
      </tem:${c.op}>
    </soapenv:Body>
  </soapenv:Envelope>"""

  def conn = new URL(endpoint).openConnection()
  conn.requestMethod = 'POST'
  conn.doOutput = true
  conn.setRequestProperty('Content-Type', 'text/xml; charset=utf-8')
  conn.setRequestProperty('SOAPAction', soapAction)
  conn.outputStream.withWriter('UTF-8') { it << body }

  def response = new XmlSlurper().parse(conn.inputStream)
  def result = response.'**'.find { it.name() == "${c.op}Result" }?.text() as Integer

  assert result == c.expected : "${c.op}(${c.a}, ${c.b}) expected ${c.expected} but got ${result}"
  log.info "OK: ${c.op}(${c.a}, ${c.b}) = ${result}"
}
log.info "All ${cases.size()} cases passed"

Run the test case. You should see 8 OK lines in the SoapUI log. Extend cases with 12 more rows to hit 20 total.

Cleaner alternative — a TestCase-level Properties step: add a Properties step, populate op, a, b, expected, then use four separate Request steps (one per operation) with ${#TestCase#a} in the body. Wrap in a DataGen loop. More clicks, but more readable in SoapUI's GUI.

Challenge solution — 2: catch a real SOAP fault

Add a new request step Divide by zero:

1. Method: POST, URL: http://www.dneonline.com/calculator.asmx
2. Body: use the Divide template with intB=0.
3. Remove the "Not SOAP Fault" assertion.
4. Add assertion: SOAP Fault.
5. Add assertion: XPath Match:

   //faultstring/text()

   Expected: Attempted to divide by zero. (exact string returned by dneonline — run the request once and copy the fault text verbatim.)

The test should pass. If the server stops returning a fault (API changes) your test will fail loudly — which is correct.

Why this matters: SOAP faults are the standard error contract. A good SOAP test suite asserts both happy-path payloads and fault shapes. A test that only checks HTTP 200 and ignores the Body is missing half the contract.

Common mistakes

• WSDL import fails with HTTP 403 or timeout

  Corporate proxies often block http://-only WSDLs. Try the URL in your browser first — if it loads, check SoapUI's preferences → HTTP Settings → Proxy. If your proxy needs auth, SoapUI has a dedicated Authentication tab per endpoint.

• XPath Match always fails even though the value looks right

  SOAP responses have namespaces. //AddResult/text() won't match because AddResult lives under the http://tempuri.org/ namespace. Declare it first: declare namespace ns='http://tempuri.org/'; //ns:AddResult/text().

• Schema Compliance fails with "WSDL not loaded"

  You're running the request against a URL, but the project's WSDL pointer went stale or was never resolved. Right-click the interface → Update Definition → re-enter the WSDL URL. Schema validation requires a parsed WSDL to compare against.

• testrunner.sh permission denied on macOS/Linux

  chmod +x /Applications/SoapUI-*.app/Contents/java/app/bin/testrunner.sh. The SoapUI installer sometimes doesn't set the execute bit on extracted binaries.

• Treating SOAP as "just XML over HTTP"

  SOAP has an entire envelope structure, namespaces, optional WS-Security, and the SOAPAction header. A Postman request hitting the .asmx endpoint will sometimes work, but assertions against the response have to know about the envelope. SoapUI hides that; Postman doesn't.