Idempotency Testing

What is idempotency?

An operation is idempotent if calling it multiple times with the same input produces the same result and has no additional side effects beyond the first call. In simpler terms: it's safe to retry.

The classic example: pressing an elevator button. Press once, the elevator comes. Press again, the elevator still comes (you didn't call two elevators). The operation is idempotent.

Non-idempotent example: withdrawing money from an ATM. Withdraw $100 once and your balance drops by $100. Withdraw $100 again and it drops another $100. The operation is not idempotent; retrying causes duplication.

Why idempotency matters

In distributed systems, network failures happen. A request might be sent but the response lost. The client doesn't know if the request succeeded, so it retries. If your system isn't idempotent:

• The same payment is processed twice (customer charged twice).
• An order is placed twice (duplicate orders).
• An email is sent twice (spam).
• A resource is created twice (duplicate records in database).

Idempotency is not just a nice property; it's essential for system reliability. Modern APIs (Stripe, AWS, Google Cloud) guarantee idempotency for critical operations. Your system should too.

Idempotent operations: safe to retry

Making non-idempotent operations safe: idempotency keys

Many operations are inherently non-idempotent (payments, orders, emails). You can't change POST's semantics. Instead, implement idempotency keys: unique identifiers that let the server recognize a retry and avoid duplicate processing.

How idempotency keys work

1. Client generates a unique identifier (UUID) for the request: Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
2. Client sends the request with this key: POST /payments with Idempotency-Key header
3. Server receives the request. Before processing, it checks: "Have I seen this key before?"
4. If no: process the payment, store the key and response in a cache/database, return the result.
5. If yes: return the cached response from the first request without processing again.
6. Network failure? Client retries with the same key. Server recognizes it and returns the same cached response.

Implementation strategies

Cache-based: Store (key → response) in Redis or in-memory cache with a TTL. Fast, good for high-throughput systems.

// Pseudocode: Redis-based idempotency
POST /payments {amount: 100}
Header: Idempotency-Key: 550e8400...

// Server-side logic
key = request.headers['Idempotency-Key']
cached = redis.get(key)
if cached:
  return cached  // Return cached response

// Process payment
result = process_payment(request.body)

// Cache the result for future retries
redis.setex(key, 3600, result)  // Cache for 1 hour
return result

Database-based: Store (key → response) in a database table. Slower but persistent across restarts. Useful for critical operations.

-- Table: idempotent_requests
CREATE TABLE idempotent_requests (
  idempotency_key UUID PRIMARY KEY,
  operation_type VARCHAR,
  request_body JSONB,
  response_body JSONB,
  status_code INT,
  created_at TIMESTAMP
);

Client-side generation

The client must generate a stable, unique key. Using a random UUID each time defeats the purpose.

// Good: stable key based on operation details
idempotencyKey = SHA256(userId + orderId + amount + timestamp)

// Better: client generates once and stores it
idempotencyKey = generateUUID()
localStorage.save('idempotency_key', idempotencyKey)

// Bad: random UUID each time (no deduplication)
idempotencyKey = generateRandomUUID()  // Different each retry!

Idempotency key design

• Unique: Each request should have a different key. Reusing keys across different operations causes collisions.
• Stable: The same logical operation (same user, same order, same payment) should use the same key. This allows safe retries.
• Immutable after creation: Once a key is sent to the server, don't change it. Changing it on retry breaks deduplication.
• Reasonable lifetime: How long should the server remember a key? Cache for hours or days (not forever). Old requests can be garbage collected.

Testing idempotency: strategies and assertions

Basic idempotency test: send request twice, get same response

@Test
void payment_isIdempotent() {
  String idempotencyKey = UUID.randomUUID().toString();

  // First request
  PaymentResponse response1 = chargeCard(
    amount: 100,
    idempotencyKey: idempotencyKey
  );
  assertEquals(200, response1.status);
  assertEquals("approved", response1.paymentStatus);
  String transactionId1 = response1.transactionId;

  // Retry with same key
  PaymentResponse response2 = chargeCard(
    amount: 100,
    idempotencyKey: idempotencyKey
  );

  // Assertion 1: same response
  assertEquals(response1.status, response2.status);
  assertEquals(response1.paymentStatus, response2.paymentStatus);

  // Assertion 2: same transaction ID (not a new charge)
  assertEquals(transactionId1, response2.transactionId);
}

Verify side effects don't duplicate

@Test
void orderCreation_isIdempotent() {
  String idempotencyKey = UUID.randomUUID().toString();

  // First request: create order
  OrderResponse response1 = createOrder(
    customerId: 123,
    items: [product1, product2],
    idempotencyKey: idempotencyKey
  );
  int orderId = response1.orderId;

  // Verify order was created
  Order order = database.getOrder(orderId);
  assertNotNull(order);
  assertEquals(2, order.items.size());

  // Retry with same key
  OrderResponse response2 = createOrder(
    customerId: 123,
    items: [product1, product2],
    idempotencyKey: idempotencyKey
  );

  // Assertion 1: same order ID
  assertEquals(orderId, response2.orderId);

  // Assertion 2: only ONE order in database (not two)
  List orders = database.getOrdersByCustomer(123);
  assertEquals(1, orders.size());

  // Assertion 3: verify only one payment record
  List payments = database.getPaymentsByOrder(orderId);
  assertEquals(1, payments.size());
}

Test error scenarios with idempotency

@Test
void payment_isIdempotent_evenWithRetryAfterError() {
  String idempotencyKey = UUID.randomUUID().toString();

  // First request: fails
  stubPaymentAPI_toReturn(500);
  assertThrows(ServerException.class, () -> {
    chargeCard(amount: 100, idempotencyKey: idempotencyKey);
  });

  // Fix the server, then retry
  stubPaymentAPI_toReturn(200);
  PaymentResponse response = chargeCard(
    amount: 100,
    idempotencyKey: idempotencyKey
  );

  // Payment should succeed on retry, not charge twice
  assertEquals("approved", response.paymentStatus);
}

Real examples: payment, order, and email idempotency

Payment processing with Stripe (industry standard)

Stripe requires idempotency_key on all charge requests. If the request times out or fails, the client retries with the same key. Stripe recognizes it and returns the result of the first attempt.

// Stripe JavaScript SDK
const charge = await stripe.paymentIntents.create(
  {
    amount: 10000,  // $100.00
    currency: 'nzd',
    description: 'Order #456',
  },
  {
    idempotencyKey: 'order-456-' + Date.now(),
  }
);

// Retry on timeout: same idempotency key
// Stripe returns the same charge, doesn't charge twice
const charge2 = await stripe.paymentIntents.create(
  {...},
  {
    idempotencyKey: 'order-456-' + storedTimestamp,  // Same key
  }
);

Order creation with duplicate detection

// POST /orders
@PostMapping("/orders")
public OrderResponse createOrder(
  @RequestBody OrderRequest request,
  @RequestHeader("Idempotency-Key") String key
) {
  // Check cache first
  OrderResponse cached = idempotencyCache.get(key);
  if (cached != null) {
    return cached;
  }

  // Process order
  Order order = new Order(request);
  order = database.save(order);

  // Charge customer
  Payment payment = paymentService.charge(
    amount: order.total,
    idempotencyKey: key + "-payment"
  );
  order.setPaymentId(payment.id);
  order = database.save(order);

  // Cache the response
  OrderResponse response = new OrderResponse(order);
  idempotencyCache.put(key, response, Duration.ofHours(1));

  return response;
}

Email sending idempotency

@Test
void emailSending_isIdempotent() {
  String idempotencyKey = "send-confirmation-order-" + orderId;

  // First send
  EmailResponse response1 = sendConfirmationEmail(
    to: "user@example.com",
    orderId: orderId,
    idempotencyKey: idempotencyKey
  );
  assertEquals("sent", response1.status);

  // Retry
  EmailResponse response2 = sendConfirmationEmail(
    to: "user@example.com",
    orderId: orderId,
    idempotencyKey: idempotencyKey
  );

  // Verify: only one email was actually sent to user
  List sent = emailService.getEmailsSentTo("user@example.com");
  assertEquals(1, sent.size());
  assertEquals("order confirmation", sent.get(0).type);
}

Distributed systems: retries and exactly-once semantics

In distributed systems, multiple services communicate over the network. Network partitions and service failures are inevitable:

• Service A sends a request to Service B. The request arrives and Service B processes it successfully. But the response is lost.
• Service A times out waiting for the response. It assumes failure and retries.
• Without idempotency, Service B processes the same request twice.

Idempotency keys, combined with coordinated state, achieve "exactly-once" delivery: the operation happens exactly once despite retries and network failures.

Database-level idempotency: unique constraints

Databases can enforce idempotency via unique constraints:

-- Prevent duplicate orders from same customer
CREATE TABLE orders (
  id UUID PRIMARY KEY,
  customer_id INT NOT NULL,
  idempotency_key VARCHAR UNIQUE NOT NULL,
  total DECIMAL,
  created_at TIMESTAMP
);

-- Unique constraint: same customer can't place two orders with same key
CREATE UNIQUE INDEX idx_customer_idempotency
ON orders(customer_id, idempotency_key);

-- Now if two requests arrive simultaneously with the same key,
-- the database enforces: only one order is created.
-- The second request gets a unique constraint violation,
-- which the application converts to "already processed, returning cached result".

API design for idempotency

If you're designing APIs, build idempotency in from the start:

• Require idempotency keys on all mutation operations: POST, PATCH, DELETE. (GET is already idempotent.)
• Document the header: "Clients MUST send an Idempotency-Key header. Requests with the same key return the cached response."
• Return the same response: If a retry arrives, return the exact same HTTP status and body as the first request.
• Cache long enough: Cache for at least hours. Stripe caches for 24 hours. This covers network timeouts and client-side retries.
• Return a hint: Optionally include a header like Idempotency-Replay: true to signal the client that this is a cached response, not a fresh request.

Message queues and event streaming: idempotency at scale

Message queues (Kafka, RabbitMQ) deliver messages at-least-once, not exactly-once. Consumers must handle duplicates:

// Kafka consumer: handle duplicate messages
consumer.subscribe(["payments.created"]);
for (record : consumer.poll()) {
  String idempotencyKey = record.headers["idempotency-key"];

  // Check: have we processed this message before?
  if (processedMessages.contains(idempotencyKey)) {
    logger.info("Duplicate message, skipping");
    continue;  // Skip processing
  }

  // Process the message
  processPayment(record);

  // Remember we processed it
  processedMessages.add(idempotencyKey);
}

Best practices

• Use universally unique identifiers (UUIDs) for keys. Ensure keys are unique across all requests. A sequential ID is not sufficient if multiple clients are generating keys.
• Test both happy path and failure scenarios. Test retries after network errors, after partial failures, after timeouts. Idempotency matters most when things go wrong.
• Implement proper cache invalidation. Don't cache forever. Set a reasonable TTL (hours or days) and clean up old entries to prevent memory leaks.
• Log idempotency key with every request. This helps debugging when customers report duplicate charges or orders. You can trace "was this request retried?"
• Coordinate idempotency keys across the call chain. If Service A calls Service B which calls Service C, propagate the idempotency key through all calls. This ensures end-to-end exactly-once semantics.
• Test with chaos engineering. Inject random failures (timeouts, 500s) into your system and verify it remains idempotent. Tools like Gremlin or Chaos Toolkit help.