Why Your APIs Need Idempotency: Your Key to Eliminating Duplicate Transactions

Why Your APIs Need Idempotency

I made many attempts to understand the concept of idempotency, reading various resources along the way. Unfortunately, each source only provided part of the picture. So, I decided to keep on reading all the resources I could find about idempotency until I fully understood the whole picture. Let’s dive deep into it:

Decoding Idempotency: More Than Just a Technical Term

I looked at the Cambridge dictionary to know what is exactly meant by Idempotency. To my surprise, I didn’t find the word in the dictionary, but I found another relative word which is Idempotent. Idempotent is not fully an English word; it has two parts: Idem, which is a Latin word and means same in English, and Potent, which means Effect or Power. So Idempotent means ‘same effect’.

In the context of APIs, an idempotent action produces the same result regardless of how many times it’s performed. For instance, an idempotent API endpoint that processes a payment for order number ‘123’ will only execute the payment once, regardless of how many times the endpoint is called with the same data.

Idempotency and HTTP Methods: A Natural Fit

Certain HTTP methods are inherently idempotent:

GET: It doesn’t modify state and return the same results for identical requests, as long as no POST requests occur on the retrieved resources.
PUT: It updates a resource once, producing the same effect for identical requests.
DELETE: It removes a resource once, regardless of how many times it’s called.

However, some HTTP methods are not naturally idempotent:

POST: It creates new resource with each call.
PATCH: It updates a resource but can yield different results each time it is applied as it can be used for appending items to an array for example or making incremental changes.

The Importance of Idempotency in Non-Idempotent Methods

Making POST or PATCH endpoints idempotent is crucial for Enabling Automatic Retries when no response is received from the server due to network glitches and request timeouts (often manifesting as status codes 502, 503, 504, and 408)

Why Can’t We Simply Retry without idempotency?

The challenge lies in the uncertainty of when the connection was lost during the request-response cycle:

The client sends the request, but the server might not receive the request at all.
The server might receive and start to process the request, but the connection is lost before completion.
The server processes the request successfully, but the connection is lost while sending the response back to the client.

In these scenarios, the client has no way of knowing whether the request was processed and retrying can cause duplicate resource creation. However, if the endpoint is idempotent, the client can safely retry the request without risking unintended side effects.

How to implement Idempotent API: A Two-Step Process

1. Client-Side:

Generate a unique key for each request.
Include this key in the request header (e.g., ‘X-Idempotency-Key’).

2. Server-Side (Check for the idempotency key):

If it’s a new key, process the request and store the response in a key-value store like Redis.
If the key exists, lookup the stored response and return it without reprocessing.

Code Snippet (This is not production-ready code; it’s for idea clarification only)

public async Task<Result<CreateOrderResponse>> ProcessRequestAsync(CreateOrderRequest request)  
{  
    if (!_httpContext.Request.Headers.TryGetValue("x-idempotency-key", out var idempotencyKey) || string.IsNullOrEmpty(idempotencyKey))  
    {  
        return Result<CreateOrderResponse>.FailureResult("Missing or invalid idempotency key.", 400);  
  
    }  
  
    string lockKey = $"{_lockPrefix}{idempotencyKey}";  
    string lockValue = Guid.NewGuid().ToString();  
    string requestKey = $"{_requestPrefix}{idempotencyKey}";  
  
    var redisConnection = await ConnectionMultiplexer.ConnectAsync("localhost:6379");  
    var db = redisConnection.GetDatabase();  
  
    // Try to acquire lock  
    if (await db.LockTakeAsync(lockKey, lockValue, _lockTimeout))  
    {  
        try  
        {  
            // Check if the request has already been processed and return cached response  
            var cachedResponse = await _cache.GetStringAsync(requestKey);  
            if (cachedResponse != null)  
            {  
                var result = JsonSerializer.Deserialize<CreateOrderResponse>(cachedResponse);  
                return Result<CreateOrderResponse>.SuccessResult(result);  
            }  
  
            var response = await ProcessOrderAsync(request);  
  
            // Store the response in Redis  
            var serializedResponse = JsonSerializer.Serialize(response);  
            await _cache.SetStringAsync(requestKey, serializedResponse, new DistributedCacheEntryOptions  
            {  
                AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(10) // Customize expiration as needed  
            });  
  
            return Result<CreateOrderResponse>.SuccessResult(response);  
        }  
        finally  
        {  
            // Release lock  
            await db.LockReleaseAsync(lockKey, lockValue);  
        }  
    }  
    else  
    {  
        // Could not acquire lock, return failure  
        return Result<CreateOrderResponse>.FailureResult("Request is being processed by another instance.", 409);  
    }  
}

Why use Redis lock in idempotent implementations?

Redis locks play a crucial role in overcoming race conditions in idempotent API implementations. Consider a scenario where:

1. The server receives an initial request and begins processing.

2. Before completion, an identical request arrives.

3. Without proper locking, the server might treat the second request as new, potentially leading to duplicate processing.

By implementing a Redis lock, we ensure that only one instance of a particular request is processed at a time. This mechanism is vital for maintaining true idempotency, especially in high-concurrency environments.

FAQs

Before concluding this article, let’s address 2 important questions that often arise when discussing idempotency in API design:

1. How often do network glitches occur?

While network glitches might seem rare, occurring in perhaps one out of a million requests, their impact becomes significant at scale. For applications handling millions of requests daily, these glitches happen to occur each day.

2. Should every endpoint be idempotent?

No, making every endpoint idempotent isn’t always necessary and can add complexity to your codebase. It’s crucial to implement idempotency where it’s most needed. One prime example is in payment APIs, where the consequences of duplicate transactions can be severe. Focus on critical operations where consistency and reliability are paramount.

Conclusion

Implementing idempotency in API design is a powerful tool for creating robust, reliable systems. While it may not be necessary for every endpoint, its application in critical areas like payment processing can significantly enhance system integrity and user trust.

Remember, the goal is to balance the need for idempotency with the complexity it adds to your system. Prioritize its implementation in areas where the consistency and reliability of operations are crucial, and you’ll build APIs that stand strong in the face of network uncertainties and high-concurrency challenges.

Why Your APIs Need Idempotency

Share this post