Performance Testing Deep Dive

1 The Hook — Why This Matters

In 2021, a major NZ bank's payment processing system began slowing down during peak business hours (9am-11am, 1pm-3pm). Payment processing took 2 seconds at 8am, 8 seconds at 10am. Customers complained. The team ran a load test: "yes, it's slow under load." A junior engineer suggested "buy more servers!" They doubled the server count. Payments were still slow. The team commissioned a performance consultant at NZD 5,000/day. Within two days, the consultant found the real bottleneck: a database query in the transaction ledger was doing a full table scan instead of using an index. The index had been accidentally dropped during a deployment. Rebuilding the index took 30 minutes and cost NZD 0. The consultant's fee could have been saved with proper performance testing and profiling.

Performance testing without profiling is expensive guesswork. Profiling without baselines is chasing shadows. You must measure, analyse, and verify fixes systematically.

2 The Rule — The One-Sentence Version

Performance testing has five layers: (1) Baselines — what is the current performance? (2) SLOs — what performance do we need? (3) Load Testing — what load causes issues? (4) Profiling — where is the bottleneck? (5) Optimisation & Verification — did the fix work? Test each layer independently, and always re-measure after changes.

3 The Analogy — Think Of It Like...

4 Watch Me Do It — Step by Step

Here is a real NZ example: an API that returns user account summaries is slow. Follow these steps to find and fix the bottleneck.

1. Establish a baseline Make 100 requests to the API in serial (one after another), measure response time. Record p50, p95, p99. Note: p50 is the median, p95 means 95% of requests are faster than this, p99 means 99% are faster. Don't use averages; they hide outliers.

   // Baseline: serial requests
   const times = [];
   for (let i = 0; i < 100; i++) {
     const start = Date.now();
     await fetch('/api/account-summary');
     times.push(Date.now() - start);
   }
   times.sort((a, b) => a - b);
   console.log({
     p50: times[50],  // median: 120ms
     p95: times[95],  // 95th percentile: 450ms
     p99: times[99]   // 99th percentile: 2100ms (outlier)
   });

   Found: p50 is 120ms (acceptable), but p99 is 2100ms (bad). This tells you the system works for 99% of users but 1% experience 2-second waits. You must investigate the p99 outliers.

2. Define SLOs (Service Level Objectives) What response time do you need? Typical SLOs for NZ fintech: p95 < 200ms, p99 < 500ms. For payment processing: p95 < 100ms, p99 < 300ms. Define SLOs based on user experience and business requirements, not what the system currently achieves.

   SLO: p95 < 200ms, p99 < 500ms. Current: p95 = 450ms, p99 = 2100ms. We're failing SLO for p95 and p99.

3. Load test to find the breaking point Gradually increase concurrent requests (1, 5, 10, 50, 100, 200 concurrent users) and measure response times. Find the point where p95 exceeds the SLO. This is your breaking point.

   // Ramp up: 10 concurrent users
   const startTime = Date.now();
   const promises = [];
   for (let i = 0; i < 10; i++) {
     promises.push(fetch('/api/account-summary'));
   }
   const results = await Promise.all(promises);
   const elapsed = Date.now() - startTime;
   console.log({concurrency: 10, avgResponseTime: elapsed / 10, p95: /* ... */});
   // Repeat with 20, 50, 100, 200 concurrent users

   Found: At 50 concurrent users, p95 jumps from 200ms to 1000ms. The system starts struggling at 50 concurrent users.

4. Profile to find the bottleneck Use CPU profiling (flame graphs), memory profiling (heap dumps), and database profiling (slow query logs). Run the API under load and capture profiles.

   // Node.js CPU profiling with clinic.js
   clinic doctor -- node app.js
   // Then run load test: artillery run load-test.yml
   // clinic.js produces a report showing which functions consume CPU

   Found: CPU profiling shows 45% of CPU time is spent in calculateAccountBalance(). Database profiling shows this function runs a query: SELECT * FROM transactions WHERE user_id = ? ORDER BY date DESC. The query takes 500ms (no index on user_id).

5. Optimize: add the missing index The transactions table has millions of rows. The query without an index does a full table scan (reads every row). Adding an index on user_id makes the query read only relevant rows.

   -- Add index
   CREATE INDEX idx_transactions_user_id ON transactions(user_id);
   -- Verify index is used
   EXPLAIN SELECT * FROM transactions WHERE user_id = ? ORDER BY date DESC;

6. Re-measure and verify improvement Run the same load test again and capture new baseline metrics. Compare p95 and p99 before and after.

   // Before: p50=120ms, p95=450ms, p99=2100ms (at 50 concurrent users, p95=1000ms)
   // After: p50=50ms, p95=120ms, p99=300ms (at 50 concurrent users, p95=150ms)
   // We now meet SLO: p95 < 200ms ✓, p99 < 500ms ✓

7. Test memory usage under load Even if response time is fast, memory might grow unbounded. Run load test for 10 minutes, monitor memory. If memory grows continuously, there's a leak.

   Pattern: Use heap dump analysis (jmap in Java, heap snapshots in Node.js) to find which objects are growing. Common causes: cached objects never evicted, event listeners never removed, database connections not closed.

8. Test cache effectiveness If your system uses caching (Redis, Memcached), measure cache hit rate. A low hit rate (< 80%) means most requests miss the cache. Optimize by increasing cache size, using better cache keys, or reducing TTL.

   // Monitor cache metrics
   const cacheHits = metrics.cacheHits;
   const cacheMisses = metrics.cacheMisses;
   const hitRate = cacheHits / (cacheHits + cacheMisses);
   console.log({hitRate: (hitRate * 100).toFixed(2) + '%'}); // Target: > 80%

9. Test for regressions in CI/CD Add performance tests to your CI/CD pipeline. On every PR, run load tests and alert if p95 degrades by > 10%. This catches performance regressions before they ship.

   // CI/CD performance gate
   baseline_p95 = 150ms  # from previous measurement
   current_p95 = run_load_test()
   if current_p95 > baseline_p95 * 1.1:  # 10% threshold
     fail("Performance regression: p95 increased from {baseline_p95}ms to {current_p95}ms")

5 When to Use It / When NOT to Use It

6 Common Mistakes — Don't Do This

7 Now You Try — Interview Warm-Up

8 Self-Check — Can You Actually Do This?

Click each question to reveal the answer. If you got all three, you're ready to own performance.

9 Interview Prep — Common Questions