Senior Automation · API Automation

Advanced API Testing

Basic GET and POST tests are junior work. Senior engineers test auth flows, validate contracts across teams, and prove APIs behave under stress. Here's how.

Senior Automation ISTQB CTAL-TAE v2.0 — Chapter 5 ~15 min read + exercise

1 The Hook — Why This Matters

A Wellington fintech team integrated with a third-party identity provider for OAuth 2.0 login. Their tests mocked the auth server perfectly. Six months later, the provider rotated their signing key without warning. Production login broke for 8,000 users. The mock-based tests had never validated the real JWT verification path. The team had 99% API test coverage — of the wrong thing.

Advanced API testing is about testing reality, not your assumptions. Mocks are tools, not substitutes. Contract tests, rate limit validation, and auth flow verification are what separate senior engineers from those who just check status codes.

2 The Rule — The One-Sentence Version

Test the contract (schema + behaviour + error responses), not just the data. Validate error paths as thoroughly as happy paths.

A test suite that only asserts status_code == 200 is not testing the API. It's pinging it. Real API testing validates schemas, auth flows, rate limits, pagination, file uploads, and every error status the API claims to return.

3 The Analogy — Think Of It Like...

Analogy

Testing a bridge by only driving a small car across the middle.

You haven't tested the weight limit (load testing), the edges (boundary cases), the emergency lanes (error paths), or what happens when a truck tries to cross (rate limiting). A bridge test that only uses a small car gives false confidence. So does an API test that only checks 200 OK.

Senior engineer insight

The most dangerous API tests are the ones that always pass. On a NZ health sector project integrating with the National Health Index, we had green CI for four months while a required NHI-Match-Mode header was silently ignored by our mock — the real API returned a completely different patient-matching behaviour when it was absent. Contract tests found this before go-live; end-to-end tests would have found it in production. Your test suite is only as trustworthy as the fidelity of your contracts.

The most common mistake: teams invest in contract tests but publish the pact file to a shared drive instead of a Pact Broker — so provider verification never runs in CI and the contract is never actually enforced.

From the field

On a project integrating with TransitNZ's TransitNZ vehicle licensing API, the team assumed the 403 Forbidden response for an expired OAuth token would carry a JSON body with a code field — the same structure as every other error. It didn't. The real provider returned plain text: "Token expired." Their error-handling middleware tried to call .json() on it, threw an unhandled exception, and the entire checkout flow crashed silently for users whose sessions had expired. The fix took 20 minutes; the discovery took three weeks of intermittent production complaints. After that, every error-path test explicitly asserted Content-Type alongside the status code, because providers don't always return what their docs claim.

4 Watch Me Do It — Step by Step

Here are four advanced API testing patterns every senior engineer should master.

Contract testing with Pact (Consumer-Driven)

const provider = new Pact({
  consumer: "OrderFrontend", provider: "OrderService", port: 1234
});

await provider.addInteraction({
  state: "order 123 exists",
  uponReceiving: "a request for order 123",
  withRequest: {
    method: "GET", path: "/orders/123",
    headers: { Authorization: like("Bearer token") }
  },
  willRespondWith: {
    status: 200,
    body: {
      id: like("123"),
      total: regex(/^\d+\.\d{2}$/, "99.99"),
      status: like("shipped")
    }
  }
});

Rate limiting validation

def test_exceeding_rate_limit_returns_429(api_client):
    limit = int(api_client.get("/api/products").headers["X-RateLimit-Limit"])
    with ThreadPoolExecutor(max_workers=limit + 5) as ex:
        futures = [ex.submit(api_client.get, "/api/products")
                   for _ in range(limit + 5)]
        statuses = [f.result().status_code for f in as_completed(futures)]
    assert 429 in statuses, "Expected at least one 429"

File upload testing

def test_upload_invalid_mime_type_blocked(api_client):
    resp = api_client.post("/api/documents",
        files={"file": ("script.exe", b"malware", "application/x-msdownload")})
    assert resp.status_code == 415

GraphQL error handling

def test_graphql_error_handling(api_client):
    resp = api_client.post("/graphql",
        json={"query": "query { user(id: null) { name } }"})
    assert resp.status_code == 200  # GraphQL returns 200 even for errors
    assert "errors" in resp.json()   # Must check response body

Contract testing styles

Style	Who writes	Best for
Provider-Driven (OpenAPI)	Provider	Public/external APIs
Consumer-Driven (Pact)	Consumer	Critical internal services
Bi-Directional	Both independently	Cross-org, no code access

Pro tip: GraphQL's convention of returning HTTP 200 even for errors catches many testers off guard. Always assert on the errors array length, not just the status code.

5 When to Use It / When NOT to Use It

✅ Use contract testing when...

Microservices communicate via APIs
Consumer and provider are on different teams
Breaking changes are expensive to fix in production

❌ Skip contract tests when...

API is internal and co-owned by one team
Schema changes are rare and well-communicated
Integration tests provide sufficient coverage

6 Common Mistakes — Don't Do This

🚫 Testing against production auth in CI

I used to think: Real auth gives the most realistic tests.
Actually: MFA blocks, rate limits, and provider downtime make tests flaky. Mock the auth server with JWT generation for CI. Run a small "auth regression" suite against the real provider weekly, not on every PR.

🚫 Schema validation = contract testing

I used to think: If the JSON matches the OpenAPI schema, the contract is valid.
Actually: Schema compliance alone misses interaction bugs. A consumer might expect a field that the schema marks as optional but the consumer requires. Contract testing validates both schema and behavioural expectations.

🚫 Ignoring error responses

I used to think: Happy path tests are enough; errors are edge cases.
Actually: Error paths are where security vulnerabilities live. Test 400, 401, 403, 404, 422, and 500 responses with correct body structures. APIs that return HTML on 500 instead of JSON are broken — and common.

7 Now You Try — Interview Warm-Up

🎯 Interactive Exercise

Scenario: Your API returns HTTP 200 for every GraphQL query, including those with invalid fields. The response body contains an errors array. Your current test only asserts status_code == 200. A production bug went undetected because the frontend didn't check the errors array.

How do you fix the test?

The fix:

Add two assertions: assert "errors" not in data or len(data["errors"]) == 0 for happy path tests, and assert "errors" in data with specific error field validation for negative tests. GraphQL's HTTP 200 convention means the status code is not a reliable success indicator. Always validate the response body structure.

Why teams fail here

Contract drift: Pact files are generated once and never updated. The consumer adds a new required field, the pact doesn't reflect it, the provider never fails verification, and a runtime 422 surfaces in production weeks later.
Auth test bypass: The CI auth token is scoped as a super-admin to avoid permission errors — so the test suite never catches a 403 that a real end-user role would hit on day one.
GraphQL optimism bias: Every assertion is status == 200 and data != null. The errors array is never inspected, so partial failures and resolver exceptions go undetected in both CI and production dashboards.
Rate limit testing skipped entirely: Teams assume the advertised rate limit works as documented. In reality, limits are often per-key, per-IP, or burst-window-based — and the behaviour under throttling (retry headers, queue depth, 429 body format) is never validated until an incident forces it.

Key takeaway

An API test suite that only validates happy paths is a liability — it gives you confidence you haven't earned, and the failures it misses always arrive at the worst possible time.

8 Self-Check — Can You Actually Do This?

Click each question to reveal the answer. If you got all three, you're ready to practice.

Q1. What is the difference between consumer-driven and provider-driven contract testing?

Consumer-driven (Pact) means the consumer defines the minimal contract and the provider verifies against it. Provider-driven means the provider publishes an OpenAPI spec and both sides validate independently. Use CDC for critical internal services; OpenAPI for public/external APIs.

Q2. Why is testing only 200 OK insufficient for API automation?

Because error paths contain security vulnerabilities, business logic bugs, and schema mismatches. A robust API suite tests every documented status code with correct response body structures.

Q3. How do you test OAuth 2.0 in CI without hitting the real identity provider?

Mock the JWKS endpoint and generate test JWTs with a private key. Configure the API under test to trust the mock issuer. Run a small suite against the real provider weekly, not on every PR.

← All Senior Automation learning Practice pages →

Advanced API Testing

1 The Hook — Why This Matters

2 The Rule — The One-Sentence Version

3 The Analogy — Think Of It Like...

4 Watch Me Do It — Step by Step

5 When to Use It / When NOT to Use It

6 Common Mistakes — Don't Do This

7 Now You Try — Interview Warm-Up

8 Self-Check — Can You Actually Do This?

Related techniques