Message Queue & Event-Driven Testing

1 The Hook — Why This Matters

In 2022, a major NZ bank processed customer transfers via Kafka. When a user transferred NZD 1000 to a friend, an event TransferRequested was published. A Ledger Service listened and recorded the transaction. A Notification Service listened and sent an SMS. Everything worked fine for six months. Then, a network partition isolated the Ledger Service from Kafka for 30 seconds. During those 30 seconds, 500 transfer events were queued. When the partition healed, the Ledger Service reprocessed all 500 events — but it had no deduplication logic. Each transfer was recorded twice. Customers saw duplicate withdrawals. Recovery took a week and cost the bank NZD 150,000 in goodwill credits.

Event-driven systems are asynchronous, distributed, and unreliable by default. Testing must account for duplicates, ordering, timeouts, and replay. A single missing test for idempotency cost millions.

2 The Rule — The One-Sentence Version

Event-driven testing spans three layers: (1) Producer Testing — does the producer emit the right message to the right topic? (2) Consumer Testing — can the consumer handle the message, duplicates, and errors? (3) End-to-End Testing — do messages flow through the entire chain, and can we recover from failures? Test each layer independently with fast unit tests, then together with integration tests using TestContainers or real brokers.

3 The Analogy — Think Of It Like...

4 Watch Me Do It — Step by Step

Here is a real NZ example: an ecommerce platform publishes OrderPlaced events. A Notification Service listens and sends emails. A Fulfillment Service listens and picks items. Test each layer.

1. Test the producer in isolation Mock the message broker. When the Order Service places an order, verify it publishes an OrderPlaced event with the correct schema (orderId, customerId, items, total). Use a test double to capture the message and assert its contents. Do not use the real broker yet.

   // Producer test (unit level)
   const mockBroker = new MockBroker();
   const orderService = new OrderService(mockBroker);
   await orderService.placeOrder({customerId: 'cust-123', items: [...]});
   expect(mockBroker.publishedMessages).toContainEqual(
     expect.objectContaining({
       type: 'OrderPlaced',
       orderId: expect.any(String),
       customerId: 'cust-123'
     })
   );

2. Test the consumer in isolation Mock the message broker. Create a OrderPlaced event and feed it to the Notification Service. Verify the service sends an email (by mocking the email client). Do not use the real broker yet.

   // Consumer test (unit level)
   const mockEmailClient = new MockEmailClient();
   const notificationService = new NotificationService(mockEmailClient);
   await notificationService.handleOrderPlaced({
     orderId: 'ord-123',
     customerId: 'cust-123',
     email: 'user@example.com'
   });
   expect(mockEmailClient.sentEmails).toContainEqual(
     expect.objectContaining({to: 'user@example.com', subject: expect.stringContaining('Order Confirmed')})
   );

3. Test idempotency (the crucial test) Feed the same message to the consumer twice. Verify it's handled idempotently: the email is sent only once, the database record is not duplicated. This is the test that the NZ bank skipped, costing NZD 150,000.

   // Idempotency test
   const notification = {
     messageId: 'msg-abc123', // unique message ID from broker
     type: 'OrderPlaced',
     orderId: 'ord-123'
   };
   await notificationService.handleOrderPlaced(notification);
   await notificationService.handleOrderPlaced(notification); // same message again
   // Verify email sent only once
   expect(mockEmailClient.sentEmails).toHaveLength(1);

4. Test exactly-once semantics (or at-least-once + deduplication) Message brokers guarantee "at-least-once" delivery, not "exactly-once." That means a message might be delivered multiple times. Your consumer must deduplicate based on messageId. Test: publish a message, simulate a network failure, republish the same message, verify the consumer handles both instances as one logical event.

   Pattern: Store messageIds in the database. Before processing a message, check if the messageId exists. If it does, skip processing. If not, process and insert the messageId. This guarantees idempotency at the cost of one database query.

5. Test dead letter queue handling When a consumer fails to process a message (e.g., email service is down), the message should be retried. After N retries, it should be moved to a Dead Letter Queue (DLQ). Test: mock the email service to fail, publish a message, verify the message is retried N times, then moved to DLQ. Verify DLQ messages can be replayed manually.

   // Consumer with DLQ
   try {
     await emailService.send(email);
   } catch (error) {
     retryCount++;
     if (retryCount > MAX_RETRIES) {
       await deadLetterQueue.send(message); // move to DLQ
     } else {
       await broker.republish(message, {delayMs: 1000 * retryCount}); // exponential backoff
     }
   }

6. Test message ordering (if applicable) Kafka guarantees ordering within a partition, but not across partitions. If your consumer processes messages out of order, does it still work? Test: publish three events (OrderPlaced, OrderShipped, OrderDelivered) to the same partition, verify they're processed in order. Then, publish to different partitions and verify you handle out-of-order gracefully (e.g., ignore DeliveryNotification if Order is not yet placed).
7. Test schema evolution The producer adds a new field (e.g., shippingAddress). The consumer doesn't expect it. Will it crash? Test: publish an event with the new field, verify the old consumer ignores it and continues working. Then, publish an event without the new field, verify the new consumer defaults the value and continues working.
8. Test end-to-end with TestContainers or real broker Spin up Kafka (or RabbitMQ) in Docker. Spin up Order Service, Notification Service, and Fulfillment Service. Place an order, verify it's published to Kafka, verify Notification Service receives it and sends an email, verify Fulfillment Service receives it and picks items. Verify all three happen asynchronously. Then, kill Notification Service, place another order, verify it's queued in Kafka, restart Notification Service, verify it processes the queued order.

   // TestContainers example
   const kafka = new KafkaContainer();
   await kafka.start();
   const orderService = new OrderService({brokerUrl: kafka.getBrokerUrl()});
   const notificationService = new NotificationService({brokerUrl: kafka.getBrokerUrl()});
   // Place order and verify both producer and consumer work
   await orderService.placeOrder({...});
   await sleep(500); // allow async processing
   expect(notificationService.handledEvents).toContainEqual(expect.objectContaining({type: 'OrderPlaced'}));

5 When to Use It / When NOT to Use It

6 Common Mistakes — Don't Do This

7 Now You Try — Interview Warm-Up

8 Self-Check — Can You Actually Do This?

Click each question to reveal the answer. If you got all three, you're ready to test async systems.

9 Interview Prep — Common Questions