Home/ Senior Automation/ Learning/ Observability & Telemetry
Senior Automation · Infrastructure

Observability & Telemetry

Stop guessing why tests failed. Learn to use distributed tracing, structured logging, and real-time metrics to diagnose complex failures in seconds instead of hours.

Senior Automation 2026 Quality Standards ~12 min read + hands-on diagnosis

1 The Hook — Why This Matters

You're running a suite of 500 API tests. One fails with a generic `500 Internal Server Error`. You check the automation logs: nothing but the status code. You check the server logs: they're a mess of thousands of entries. It takes you three hours to track down that a downstream microservice timed out because of a specific database lock. This is waste.

In a senior role, your value isn't just writing the test; it's reducing the Mean Time to Diagnosis (MTTD). With distributed tracing, that same failure would show you the exact hop where the error occurred in milliseconds. Observability turns "I think it's broken" into "I know exactly why it's broken."

2 The Rule — The One-Sentence Version

A test without a correlation ID is just a guess; true automation stability is built on high-cardinality telemetry, not just pass/fail assertions.

Every automated request must carry a unique header (e.g., `X-Test-ID`) that links the automation execution to the backend traces, logs, and metrics.

3 The Analogy — Think Of It Like...

Analogy

From a Car's "Check Engine" Light to a Full Diagnostic Scanner.

A traditional test result is like a "Check Engine" light: you know something is wrong, but you don't know what. Observability is the mechanic's scanner: it tells you the exact cylinder that's misfiring, the fuel pressure at the moment of failure, and the history of the part. Don't settle for a light; get the scanner.

Senior engineer insight

The biggest mindset shift I made was treating test telemetry as a first-class product deliverable, not an afterthought. On a payments integration project with a major NZ bank, we instrumented every automated request with X-Test-Run-ID and X-Scenario-Name headers from day one — and when a settlement reconciliation test started failing intermittently at midnight, we had a direct link from the assertion failure straight to a database deadlock between the batch job and our test data teardown. Without those headers, we would have spent days guessing. The test didn't change; the system did, and we could prove it.

The most common mistake: teams instrument their test framework beautifully but never confirm the backend actually propagates the correlation ID beyond the first service hop — so the trace snaps after the gateway and you're blind for the rest of the call chain.

4 Watch Me Do It — Step by Step

Here is how a Senior Engineer implements the "Observability-Driven Development" (ODD) pattern.

  1. Inject Trace Context Headers Configure your HTTP client (Playwright, Axios, RestAssured) to inject a `traceparent` header into every request. This ensures the backend knows this request is part of a specific test run.
  2. Correlate Logs with Span IDs Ensure your automation framework logs the Trace ID for every failed step. When a test fails, the first line of the error report should be a link to the trace in Jaeger or Honeycomb.
  3. Monitor Performance via OTel Metrics Don't just check if the page loaded; check the `http.server.duration` metric for that specific request. If the test passes but took 10x longer than usual, your automation should trigger a performance warning.
  4. Use "High-Cardinality" Metadata Attach tags like `test.name`, `test.environment`, and `git.commit_sha` to your telemetry. This allows you to answer complex questions like "Which commit caused the latency spike in the checkout API?"
Senior Tip: In NZ, where we often work with smaller, integrated teams, you have the unique opportunity to pair with DevOps/SREs to define these standards. Don't wait for them to give you access; ask for the API endpoints to query their telemetry.

From the field

A QA team at a TransitNZ supplier was running overnight Playwright suites against a road-event data ingestion API. Tests passed 95% of the time, and the 5% failures were written off as "environment flakiness." The assumption was that a passing HTTP 200 meant data was correctly ingested. When they finally wired OpenTelemetry spans into the test runner and pushed trace context through to the backend, they discovered that roughly 1 in 20 payloads was being silently dropped by a message queue that was at capacity — the API accepted the request and returned 200, but the consumer was full and discarding events without error. The data loss had been happening for six weeks. The lesson: a 200 response is a contract about the HTTP layer, not about what the downstream system did with your data. You need traces to see past the surface.

5 When to Use It / When NOT to Use It

✅ Prioritise Telemetry when...

  • Testing microservice architectures
  • Debugging intermittent "flaky" failures
  • Automating complex multi-step state changes
  • Scaling to thousands of daily executions

❌ Simple logging is enough for...

  • Static frontend-only applications
  • Single-monolith legacy systems with no OTel support
  • One-off scripts and internal utilities
  • Simple UI "smoke" checks

6 Common Mistakes — Don't Do This

🚫 "Logging" everything as text

I used to think: More log lines equal more info.
Actually: Text logs are expensive to search and hard to aggregate. Use Structured Logging (JSON). This allows you to filter by specific fields (e.g., `user_id`, `status_code`) across millions of lines in milliseconds.

🚫 Ignoring "Cardinality Explosions"

I used to think: Put every possible variable into a tag.
Actually: Some telemetry systems charge by "cardinality" (the number of unique tag values). Putting a timestamp in a tag can cost thousands of dollars. Only tag fields that you actually need to group or filter by.

🚫 Correlation IDs that don't propagate

I used to think: I'll generate a trace ID in the test, and the backend will automatically connect it.
Actually: The backend won't know about your trace unless you explicitly pass it in a header (e.g., `traceparent`, `X-Trace-ID`). The entire request chain must propagate it. If any hop drops it, the chain breaks.

When this technique fails

You've instrumented your tests with beautiful telemetry. But the backend doesn't export traces to your observability platform, or the platform's retention is only 7 days and you need to debug a flaky test from last month. Telemetry is only useful if someone can act on it. Before building, ensure the backend exports what you need, and you have a tool/dashboard to query it.

7 Now You Try — Hands-on Diagnosis

🎯 Interactive Exercise

Scenario: You are investigating a flaky test that fails 1 in 10 times during the "Add to Cart" step. The automation log says `Timeout: Element not found`.

What are the first three things you check in your observability dashboard?

  1. The Distributed Trace: Did the backend `/cart` API actually return a 200, or did a downstream service fail?
  2. Database Metrics: Was there a latency spike or lock contention in the database at the exact second the test failed?
  3. Pod Health (CPU/Memory): Was the specific instance of the service being hammered by another test or a background job?

Why teams fail here

  • Treating observability as an ops concern, not a QA concern — testers wait for the SRE team to build dashboards instead of pushing trace context from their own automation; by the time production observability is mature, test runs still emit no structured telemetry and the two worlds never connect.
  • Correlation IDs that die at the load balancer — the test injects a traceparent header, but the nginx or F5 config strips custom headers before the request reaches the app; nobody verifies propagation end-to-end, so every trace is a stub.
  • Logging at INFO level in non-prod environments only — test environments are often configured with verbose logging suppressed to reduce noise, which means the one environment where tests run is also the one with the least diagnostic signal when things break.
  • Cardinality abuse in short-lived test environments — injecting test.run_timestamp or unique UUID tags on every metric causes cardinality explosions in platforms like Prometheus or Datadog; at scale (think Revenue NZ batch processing suites running hundreds of parameterised scenarios), this can cost thousands of dollars a month and get observability access revoked for the QA team entirely.

Key takeaway

A test that fails without a trace is an anecdote; a test that fails with a correlated span, structured log, and a metric timestamp is evidence — and evidence is what separates senior engineers from everyone else when production is on fire at 2 AM.

8 Self-Check — Can You Actually Do This?

Q1. What is the difference between Monitoring and Observability?

Monitoring tells you *if* a system is failing (the symptoms). Observability tells you *why* it is failing (the root cause) by allowing you to ask questions you didn't predict.

Q2. What are the "Three Pillars" of Observability?

Logs (records of events), Metrics (numerical data over time), and Traces (the journey of a request through services).

Q3. How does a Senior Automation Engineer reduce MTTD?

By injecting correlation IDs into every automated request, enabling instant linking from a test failure to the exact backend trace and logs.

Related techniques

  • Chaos Engineering — observability validates chaos experiments — without telemetry, you cannot confirm what happened during failure injection
  • Performance Testing — performance testing and observability are inseparable — telemetry provides the metrics that performance tests validate
  • Defect Management — observability data is defect evidence — production defects are reproduced and diagnosed through telemetry

9 Interview Prep — Senior Q&A

SREs and platform engineers ask about real production debugging. These are typical Auckland and Welling ton DevOps interview questions.

Q. "Your test failed with 'Connection Timeout.' How do you quickly determine if it's an app bug or infrastructure?"

With telemetry: Pull the test's trace ID from the logs. Query the backend trace for the same ID. If the backend received the request but got stuck (open span with no close), it's a backend bug. If the backend never received the request (no trace), it's network/infrastructure. If the request reached the backend and completed, but the test timed out, it's a test logic issue (wrong selector, assertion). Correlation IDs answer this in seconds instead of hours.

Q. "Your test dashboard shows a 'spike' in latency at 2 AM every night. Tests still pass. Is this a problem?"

Absolutely. Passing tests don't mean the system is healthy. Query the backend metrics for that 2 AM window. Check: Are database backups running? Is Kubernetes doing node maintenance? Is cache missing due to eviction policy? Spikes in latency, even under SLO, are signals. Instrument your tests to query actual metrics (p99, error budget burn rate) and alert if they deviate from baseline.

Q. "What information should you capture in a test failure that helps you debug faster?"

1) The test execution ID (UUID), 2) The correlation trace ID from backend logs, 3) Timestamp of failure (ISO 8601), 4) The assertion that failed (not just line number), 5) Environment (staging/prod), 6) A link to the trace in your observability platform (e.g., Jaeger or Honeycomb). Optionally: HAR file (HTTP Archive) of the test run, browser console logs, backend error logs. The goal: someone can immediately query the trace and reconstruct what happened without re-running the test.

Q. "Your backend team says 'We don't emit traces yet.' Should you skip telemetry?"

No. Start with what you can control: 1) Add correlation ID headers to every request your tests make. 2) Log the correlation ID when a test fails. 3) Ask the backend team to echo the header in their response (so you know they received it). 4) Once they start emitting traces, your tests are ready to consume them. You're building the bridge incrementally instead of waiting for them to build the whole thing.

← All Senior learning Practice pages →