CI/CD Integration
CI/CD is not just about running tests. It's about deciding, automatically, whether code is safe to deploy. Learn how to design gates that actually protect production.
1 The Hook — Why This Matters
In 2024, a NZ insurance company deployed a pricing engine update that passed all 800 automated tests. It also caused a 40-minute outage affecting 12,000 customers. The tests were green because none of them validated performance under load, and the deployment went straight to 100% of production servers. There was no canary, no rollback automation, and no SLO-based gate.
Running tests in CI is table stakes. The senior engineer's job is deciding what happens next: deploy, hold, or rollback. That decision must be automated, evidence-based, and irreversible by human override.
2 The Rule — The One-Sentence Version
Every deployment pipeline must have automated gates that can halt or rollback a release without human intervention, based on test results and production metrics.
Manual go/no-go decisions are too slow for modern delivery and too vulnerable to pressure. "The CEO wants this live by Friday" should not be able to bypass a failing quality gate.
3 The Analogy — Think Of It Like...
A airport security checkpoint with automatic doors.
The metal detector scans passengers (tests). If it beeps, the door stays closed (gate blocks deploy). If it's silent, the door opens automatically (deploy proceeds). There is no security guard you can sweet-talk into letting you through. The system's decision is final, fast, and based on evidence. That's what a quality gate does for code.
Senior engineer insight
The biggest shift in how I think about CI/CD came when I stopped asking "do the tests pass?" and started asking "does passing prove the system is safe to deploy?" At TransitNZ we had a 98% pass rate that told us nothing about real-world behaviour, because every test was written against mocked dependencies — the pipeline was green on the day an integration broke production for four hours. The gate has to be testing the things that actually fail in production: downstream service contracts, latency at realistic concurrency, and data-layer integrity across migration boundaries.
The most common mistake: teams instrument their pipeline with gates but never validate that a gate would actually catch a real defect — the gate exists but has never been seen to block a deploy.
From the field
A NZ bank running core banking modernisation assumed their canary deployment was protecting production: 5% traffic routed to the new service, automated rollback wired up, dashboards live. What they discovered three months in was that the 5% slice never included authenticated business-banking sessions — the routing rule split on a session cookie that corporate users didn't carry. So the canary was only ever exercising anonymous traffic, and a defect in the business payment flow was invisible to every health check. They found out when a relationship manager called to say business customers couldn't submit same-day payments. The lesson: canary routing logic must be validated as carefully as the application under test — treat it as a testable artefact, not infrastructure plumbing.
4 Watch Me Do It — Step by Step
Here is how to design a production-quality deployment pipeline.
- Define your quality gates
Gate Type Criteria Example Functional Pass rate >= 99% Zero critical defects; contract tests green Performance p95 latency <= budget API < 500ms; error rate < 0.5% Reliability Flake rate <= 1% Suite time <= 10 min for PR gates Security Zero critical CVEs SAST findings resolved or risk-accepted - Implement a canary deployment with automated rollback
deploy: needs: [test, security-scan] steps: - name: Deploy to Blue run: kubectl set image deployment/app-blue app=app:v$GITHUB_SHA - name: Smoke Test Blue run: pytest tests/smoke --base-url=https://blue.example.com - name: Route 10% Traffic to Blue run: istioctl proxy-config route shift --blue=10 - name: Canary Analysis (5 min) run: k6 run tests/canary.js --env TARGET=blue.example.com - name: Full Cutover or Rollback run: | if [ $? -eq 0 ]; then istioctl proxy-config route shift --blue=100 else istioctl proxy-config route shift --blue=0 echo "Canary failed -- rolled back" - Track DORA metrics
Metric Elite Target Deployment Frequency Multiple times per day Lead Time for Changes < 1 hour Mean Time to Recovery < 1 hour Change Failure Rate < 5%
5 When to Use It / When NOT to Use It
✅ Use canary deployments when...
- Releasing high-risk changes
- You have production SLOs to protect
- Rollback must happen in < 60 seconds
- Traffic can be split by percentage
❌ Skip canary when...
- Deploying to internal-only tools
- The change is a hotfix with known good state
- Infrastructure doesn't support traffic splitting
6 Common Mistakes — Don't Do This
🚫 Slow E2E as PR gate
I used to think: More gates mean more safety.
Actually: A 40-minute PR pipeline trains developers to bypass it. Run fast unit/integration gates on PR; schedule heavy E2E nightly. Safety that developers circumvent is not safety.
🚫 Manual rollbacks
I used to think: Rollback is a decision for the on-call engineer.
Actually: MTTR is measured in minutes, not meetings. Automated rollback on canary KPI breach (error rate, latency) is the only way to hit <1 hour recovery. The human's job is investigating after the system is already safe.
🚫 No artifact retention policy
I used to think: We should keep every build artifact forever.
Actually: Artifact storage costs compound. Retain JUnit XML and screenshots for 30 days, HTML reports for 7 days, and full video traces for 3 days. Archive release artifacts (container images) indefinitely.
7 Now You Try — Interview Warm-Up
Scenario: Your E2E suite has a 15% flake rate. Management wants to mandate that all E2E tests must pass before any deployment. The team is spending 30% of their time rerunning flaky tests.
What is your three-step remediation plan?
Three steps:
- Quarantine: Move all flaky tests to a separate "quarantine" suite that runs nightly but does not block deployment. Require each quarantined test to have a Jira ticket with an owner and deadline.
- Root cause: Analyse the top 10 flaky tests. Common fixes: explicit waits instead of sleeps, deterministic test data, pinned browser versions, and test isolation.
- Governance: Enforce a flake-rate SLO (e.g., <2%). Any test that flakes more than twice in a week gets quarantined automatically. No exceptions.
Why teams fail here
- Sharding without shard-level reporting: teams split their suite across 10 runners to cut time from 40 minutes to 4, then realise the merged JUnit report drops the shard context — when a test flakes on shard 7 only, they can't reproduce it locally and spend longer debugging than they saved parallelising.
- Environment promotion gates that check the wrong environment: the UAT gate passes because the smoke test hits UAT's own stub service, not the real downstream — production has a different API version and the defect was never exposed. Revenue NZ's PAYE integration history is full of this pattern.
- Rollback that isn't tested until it's needed: automated rollback scripts age unexercised, pick up drift from infrastructure changes, and fail silently or partially at 2am on the night of an actual incident — often leaving the system in a split-brain state worse than either version.
- DORA metrics tracked but not actioned: teams instrument deployment frequency and change failure rate, present them in quarterly reviews, and never close the loop — a 12% change failure rate sits on a slide for six months while the same pipeline issues cause the same incidents, because no-one owns the metric as a system property to improve.
Key takeaway
A quality gate that has never blocked a deployment is not a gate — it's a sign that lights up green while the door stays permanently open.
8 Self-Check — Can You Actually Do This?
Click each question to reveal the answer. If you got all three, you're ready to practice.
Q1. What is the difference between blue-green and canary deployment?
Blue-green deploys to an identical environment and switches traffic instantly. Good for fast rollback. Canary gradually shifts traffic (5% -> 25% -> 100%) with health checks at each stage. Good for catching issues at small scale before they affect everyone.
Q2. Why should quality gates be automated rather than manual?
Manual gates are too slow for modern delivery and vulnerable to organisational pressure ("the CEO wants it live by Friday"). Automated gates enforce policy consistently and provide an audit trail.
Q3. What four DORA metrics should a senior automation engineer track?
Deployment Frequency, Lead Time for Changes, Mean Time to Recovery (MTTR), and Change Failure Rate. These measure both speed and stability of delivery.