Idempotency Testing

What is idempotency?

An operation is idempotent if calling it multiple times with the same input produces the same result and has no additional side effects beyond the first call. In simpler terms: it's safe to retry.

The classic example: pressing an elevator button. Press once, the elevator comes. Press again, the elevator still comes (you didn't call two elevators). The operation is idempotent.

Non-idempotent example: withdrawing money from an ATM. Withdraw $100 once and your balance drops by $100. Withdraw $100 again and it drops another $100. The operation is not idempotent; retrying causes duplication.

Why idempotency matters

In distributed systems, network failures happen. A request might be sent but the response lost. The client doesn't know if the request succeeded, so it retries. If your system isn't idempotent:

• The same payment is processed twice (customer charged twice).
• An order is placed twice (duplicate orders).
• An email is sent twice (spam).
• A resource is created twice (duplicate records in database).

Idempotency is not just a nice property; it's essential for system reliability. Modern APIs (Stripe, AWS, Google Cloud) guarantee idempotency for critical operations. Your system should too.

Idempotent operations: safe to retry

Making non-idempotent operations safe: idempotency keys

Many operations are inherently non-idempotent (payments, orders, emails). You can't change POST's semantics. Instead, implement idempotency keys: unique identifiers that let the server recognize a retry and avoid duplicate processing.

How idempotency keys work

1. Client generates a unique identifier (UUID) for the request: Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
2. Client sends the request with this key: POST /payments with Idempotency-Key header
3. Server receives the request. Before processing, it checks: "Have I seen this key before?"
4. If no: process the payment, store the key and response in a cache/database, return the result.
5. If yes: return the cached response from the first request without processing again.
6. Network failure? Client retries with the same key. Server recognizes it and returns the same cached response.

Implementation strategies

Cache-based: Store (key → response) in Redis or in-memory cache with a TTL. Fast, good for high-throughput systems.

// Pseudocode: Redis-based idempotency
POST /payments {amount: 100}
Header: Idempotency-Key: 550e8400...

// Server-side logic
key = request.headers['Idempotency-Key']
cached = redis.get(key)
if cached:
  return cached  // Return cached response

// Process payment
result = process_payment(request.body)

// Cache the result for future retries
redis.setex(key, 3600, result)  // Cache for 1 hour
return result

Database-based: Store (key → response) in a database table. Slower but persistent across restarts. Useful for critical operations.

-- Table: idempotent_requests
CREATE TABLE idempotent_requests (
  idempotency_key UUID PRIMARY KEY,
  operation_type VARCHAR,
  request_body JSONB,
  response_body JSONB,
  status_code INT,
  created_at TIMESTAMP
);

Client-side generation

The client must generate a stable, unique key. Using a random UUID each time defeats the purpose.

// Good: stable key based on operation details
idempotencyKey = SHA256(userId + orderId + amount + timestamp)

// Better: client generates once and stores it
idempotencyKey = generateUUID()
localStorage.save('idempotency_key', idempotencyKey)

// Bad: random UUID each time (no deduplication)
idempotencyKey = generateRandomUUID()  // Different each retry!

Idempotency key design

• Unique: Each request should have a different key. Reusing keys across different operations causes collisions.
• Stable: The same logical operation (same user, same order, same payment) should use the same key. This allows safe retries.
• Immutable after creation: Once a key is sent to the server, don't change it. Changing it on retry breaks deduplication.
• Reasonable lifetime: How long should the server remember a key? Cache for hours or days (not forever). Old requests can be garbage collected.

Testing idempotency: strategies and assertions

Basic idempotency test: send request twice, get same response

@Test
void payment_isIdempotent() {
  String idempotencyKey = UUID.randomUUID().toString();

  // First request
  PaymentResponse response1 = chargeCard(
    amount: 100,
    idempotencyKey: idempotencyKey
  );
  assertEquals(200, response1.status);
  assertEquals("approved", response1.paymentStatus);
  String transactionId1 = response1.transactionId;

  // Retry with same key
  PaymentResponse response2 = chargeCard(
    amount: 100,
    idempotencyKey: idempotencyKey
  );

  // Assertion 1: same response
  assertEquals(response1.status, response2.status);
  assertEquals(response1.paymentStatus, response2.paymentStatus);

  // Assertion 2: same transaction ID (not a new charge)
  assertEquals(transactionId1, response2.transactionId);
}

Verify side effects don't duplicate

@Test
void orderCreation_isIdempotent() {
  String idempotencyKey = UUID.randomUUID().toString();

  // First request: create order
  OrderResponse response1 = createOrder(
    customerId: 123,
    items: [product1, product2],
    idempotencyKey: idempotencyKey
  );
  int orderId = response1.orderId;

  // Verify order was created
  Order order = database.getOrder(orderId);
  assertNotNull(order);
  assertEquals(2, order.items.size());

  // Retry with same key
  OrderResponse response2 = createOrder(
    customerId: 123,
    items: [product1, product2],
    idempotencyKey: idempotencyKey
  );

  // Assertion 1: same order ID
  assertEquals(orderId, response2.orderId);

  // Assertion 2: only ONE order in database (not two)
  List orders = database.getOrdersByCustomer(123);
  assertEquals(1, orders.size());

  // Assertion 3: verify only one payment record
  List payments = database.getPaymentsByOrder(orderId);
  assertEquals(1, payments.size());
}

Test error scenarios with idempotency

@Test
void payment_isIdempotent_evenWithRetryAfterError() {
  String idempotencyKey = UUID.randomUUID().toString();

  // First request: fails
  stubPaymentAPI_toReturn(500);
  assertThrows(ServerException.class, () -> {
    chargeCard(amount: 100, idempotencyKey: idempotencyKey);
  });

  // Fix the server, then retry
  stubPaymentAPI_toReturn(200);
  PaymentResponse response = chargeCard(
    amount: 100,
    idempotencyKey: idempotencyKey
  );

  // Payment should succeed on retry, not charge twice
  assertEquals("approved", response.paymentStatus);
}

Real examples: payment, order, and email idempotency

Payment processing with Stripe (industry standard)

Stripe requires idempotency_key on all charge requests. If the request times out or fails, the client retries with the same key. Stripe recognizes it and returns the result of the first attempt.

// Stripe JavaScript SDK
const charge = await stripe.paymentIntents.create(
  {
    amount: 10000,  // $100.00
    currency: 'nzd',
    description: 'Order #456',
  },
  {
    idempotencyKey: 'order-456-' + Date.now(),
  }
);

// Retry on timeout: same idempotency key
// Stripe returns the same charge, doesn't charge twice
const charge2 = await stripe.paymentIntents.create(
  {...},
  {
    idempotencyKey: 'order-456-' + storedTimestamp,  // Same key
  }
);

Order creation with duplicate detection

// POST /orders
@PostMapping("/orders")
public OrderResponse createOrder(
  @RequestBody OrderRequest request,
  @RequestHeader("Idempotency-Key") String key
) {
  // Check cache first
  OrderResponse cached = idempotencyCache.get(key);
  if (cached != null) {
    return cached;
  }

  // Process order
  Order order = new Order(request);
  order = database.save(order);

  // Charge customer
  Payment payment = paymentService.charge(
    amount: order.total,
    idempotencyKey: key + "-payment"
  );
  order.setPaymentId(payment.id);
  order = database.save(order);

  // Cache the response
  OrderResponse response = new OrderResponse(order);
  idempotencyCache.put(key, response, Duration.ofHours(1));

  return response;
}

Email sending idempotency

@Test
void emailSending_isIdempotent() {
  String idempotencyKey = "send-confirmation-order-" + orderId;

  // First send
  EmailResponse response1 = sendConfirmationEmail(
    to: "user@example.com",
    orderId: orderId,
    idempotencyKey: idempotencyKey
  );
  assertEquals("sent", response1.status);

  // Retry
  EmailResponse response2 = sendConfirmationEmail(
    to: "user@example.com",
    orderId: orderId,
    idempotencyKey: idempotencyKey
  );

  // Verify: only one email was actually sent to user
  List sent = emailService.getEmailsSentTo("user@example.com");
  assertEquals(1, sent.size());
  assertEquals("order confirmation", sent.get(0).type);
}

Distributed systems: retries and exactly-once semantics

In distributed systems, multiple services communicate over the network. Network partitions and service failures are inevitable:

• Service A sends a request to Service B. The request arrives and Service B processes it successfully. But the response is lost.
• Service A times out waiting for the response. It assumes failure and retries.
• Without idempotency, Service B processes the same request twice.

Idempotency keys, combined with coordinated state, achieve "exactly-once" delivery: the operation happens exactly once despite retries and network failures.

Database-level idempotency: unique constraints

Databases can enforce idempotency via unique constraints:

-- Prevent duplicate orders from same customer
CREATE TABLE orders (
  id UUID PRIMARY KEY,
  customer_id INT NOT NULL,
  idempotency_key VARCHAR UNIQUE NOT NULL,
  total DECIMAL,
  created_at TIMESTAMP
);

-- Unique constraint: same customer can't place two orders with same key
CREATE UNIQUE INDEX idx_customer_idempotency
ON orders(customer_id, idempotency_key);

-- Now if two requests arrive simultaneously with the same key,
-- the database enforces: only one order is created.
-- The second request gets a unique constraint violation,
-- which the application converts to "already processed, returning cached result".

API design for idempotency

If you're designing APIs, build idempotency in from the start:

• Require idempotency keys on all mutation operations: POST, PATCH, DELETE. (GET is already idempotent.)
• Document the header: "Clients MUST send an Idempotency-Key header. Requests with the same key return the cached response."
• Return the same response: If a retry arrives, return the exact same HTTP status and body as the first request.
• Cache long enough: Cache for at least hours. Stripe caches for 24 hours. This covers network timeouts and client-side retries.
• Return a hint: Optionally include a header like Idempotency-Replay: true to signal the client that this is a cached response, not a fresh request.

Message queues and event streaming: idempotency at scale

Message queues (Kafka, RabbitMQ) deliver messages at-least-once, not exactly-once. Consumers must handle duplicates:

// Kafka consumer: handle duplicate messages
consumer.subscribe(["payments.created"]);
for (record : consumer.poll()) {
  String idempotencyKey = record.headers["idempotency-key"];

  // Check: have we processed this message before?
  if (processedMessages.contains(idempotencyKey)) {
    logger.info("Duplicate message, skipping");
    continue;  // Skip processing
  }

  // Process the message
  processPayment(record);

  // Remember we processed it
  processedMessages.add(idempotencyKey);
}

Context guide

How the right level of idempotency testing effort changes based on project context.

Trade-offs

What you gain and what you give up when you choose idempotency testing.

Best practices

• Use universally unique identifiers (UUIDs) for keys. Ensure keys are unique across all requests. A sequential ID is not sufficient if multiple clients are generating keys.
• Test both happy path and failure scenarios. Test retries after network errors, after partial failures, after timeouts. Idempotency matters most when things go wrong.
• Implement proper cache invalidation. Don't cache forever. Set a reasonable TTL (hours or days) and clean up old entries to prevent memory leaks.
• Log idempotency key with every request. This helps debugging when customers report duplicate charges or orders. You can trace "was this request retried?"
• Coordinate idempotency keys across the call chain. If Service A calls Service B which calls Service C, propagate the idempotency key through all calls. This ensures end-to-end exactly-once semantics.
• Test with chaos engineering. Inject random failures (timeouts, 500s) into your system and verify it remains idempotent. Tools like Gremlin or Chaos Toolkit help.

(a) GET balance — Idempotent. Reading data changes nothing; calling it 10 times returns the same value with no side effects.

(b) POST transfer — NOT idempotent as-is. Each call moves another $500. A retry after a lost response moves the money twice. Make it safe with an idempotency key so the server recognises the retry and performs the transfer once.

(c) PUT profile — Idempotent. PUT sets the resource to a given state; applying "phone = 021..." twice leaves the same final state. Retrying is safe.

(d) POST billpay — NOT idempotent as-is. Each call pays the bill again, so a retry double-pays the power company. Needs an idempotency key (or a server-side duplicate check) so the bill is paid exactly once.

The pattern: reads (GET) and full replacements (PUT) are naturally idempotent; creates/charges (POST) usually are not and need an idempotency key.

Why it still double-charges: the key is generated fresh inside payOnce() on every call, so the retry sends a DIFFERENT idempotency key. The server has no way to recognise the second request as a retry of the first — it looks like a brand-new payment, so it charges again. A random key per attempt defeats the entire purpose.

The fix: generate the key ONCE for the logical operation and reuse the SAME key on every retry.

  // Generate once, before the first attempt
  var key = generateUUID();
  localStorage.save('paymentKey', key);

  function payWithRetry() {
    return chargeCard({ amount: 9000, idempotencyKey: key }); // same key every time
  }
  // On timeout, call payWithRetry() again — same key, so the server returns the first result.

Key design rules this satisfies: the key is stable across retries, immutable once sent, and unique per logical operation. The server stores key -> response and returns the cached response for any repeat of that key.

Test name: orderCreation_isIdempotent_onRetry

Setup: a known customer; generate ONE idempotency key for the order.

Steps:
1. Send POST /orders with the items and the Idempotency-Key header. Capture response1.
2. Send the IDENTICAL request again with the SAME Idempotency-Key (simulating a retry after a lost response).

Assertions on the HTTP responses:
- response1 status is success (e.g. 201).
- response2 has the same status and the SAME orderId as response1 (not a new order).

Assertions on the database / side effects (the important part):
- Exactly ONE order exists for that idempotency key / customer (count == 1, not 2).
- Exactly ONE payment record for that order (the card was charged once).
- Only ONE confirmation email/notification was queued.

A failure-scenario variant worth adding: make the first attempt fail mid-way (stub the payment service to return 500), then retry with the same key after "fixing" the service. Assert the order is created and the card charged exactly once — idempotency matters most when the first attempt half-failed. Checking the DB side effects, not just the HTTP status, is what separates a real idempotency test from a shallow one.

How this has changed

The field moved. Here is how Idempotency Testing evolved from its origins to current practice.

Pre-2010

Idempotency is a mathematical and API design concept but not a recognised testing technique. Developers assume correct idempotent behaviour; testers verify happy-path outcomes. Double-submission and retry bugs are common and often discovered in production.

2010

REST API design principles popularised by Roy Fielding's dissertation gain widespread adoption. HTTP PUT, DELETE, and GET are defined as idempotent; POST is not. Idempotency becomes an explicit API design criterion, creating testable expectations.

2014

Stripe introduces idempotency keys for payment APIs. The pattern — clients supply a unique key so retries return the same result without creating duplicates — becomes standard for payment and financial APIs. Testing idempotency keys becomes a required test category for payment integrations.

2018

Distributed systems complexity increases retry complexity. Kafka consumer groups, SQS message processing, and microservice event handling all require idempotency testing. Chaos engineering experiments surface idempotency failures under real network conditions.

Now

Idempotency testing is a first-class concern for any API that handles financial transactions, state changes, or operations that must not be duplicated. NZ payment systems (EFTPOS, bank transfer APIs) require idempotency testing as a regulatory expectation for operational resilience.

Related techniques

• API Testing — idempotency is a critical API property — test it at the API layer with duplicate requests
• State Transition Testing — idempotency failures often manifest as incorrect state transitions on duplicate calls
• Chaos Engineering — chaos experiments that simulate network retries reveal idempotency violations in production conditions

Interview Questions

What NZ hiring managers ask about Idempotency Testing — and what strong answers look like.

Why is idempotency important in payment APIs, and how do you test it?

Strong answer: Idempotency ensures that retrying a request (due to network failure, timeout, or client error) does not create duplicate operations. For payments, a non-idempotent retry means a customer is charged twice. To test: send a payment request with an idempotency key, receive a success response, then send the identical request again with the same key. Verify the second response returns the same result as the first (HTTP 200 with the same transaction ID) without creating a second charge. Also test: the same request with a different key should create a new charge; a different request with the same key should be rejected.

Junior/Mid

A retry storm hits your API after a brief outage. How does idempotency protect users, and what do you need to test to verify that protection?

Strong answer: During a retry storm, clients that timed out all retry simultaneously. Without idempotency, each retry creates a new operation — users could be charged or enrolled many times. With idempotency keys, the server recognises a retry and returns the original result without re-executing. To verify: simulate a network partition between client and server (client sends, server processes but client never receives 200), then retry with the same key. Verify the server returns the original result without re-processing. Also test key expiry: what happens when a key is retried after the idempotency window expires? And verify that the idempotency store is consistent under concurrent requests with the same key.

Senior/Lead

Self-Check