How to Implement API Request Deduplication

Understanding the Problem

When building an API, handling duplicate requests efficiently is crucial. Duplicate requests can arise due to network retries, client-side errors, or intentional abuse. These can lead to resource exhaustion, inconsistent data states, and degraded performance. To mitigate these issues, implementing request deduplication is essential.

Solution with Code

The solution involves maintaining an idempotency key for each request. This key helps identify and filter out duplicate requests, ensuring each unique request is processed only once.

Step-by-Step Implementation

1. Client-Side Setup: Ensure each API request includes a unique idempotency-key in the headers.

function generateIdempotencyKey() {
  return `${new Date().getTime()}-${Math.random().toString(36).substr(2, 9)}`;
}

async function makeApiRequest(endpoint, data) {
  const idempotencyKey = generateIdempotencyKey();
  const response = await fetch(endpoint, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Idempotency-Key': idempotencyKey
    },
    body: JSON.stringify(data)
  });
  return response.json();
}

2. Server-Side Setup: Implement a middleware to check and store idempotency keys.

const express = require('express');
const app = express();
const bodyParser = require('body-parser');
const deduplicationCache = new Map();

app.use(bodyParser.json());

function deduplicationMiddleware(req, res, next) {
  const idempotencyKey = req.headers['idempotency-key'];
  if (!idempotencyKey) {
    return res.status(400).send('Idempotency key is required.');
  }

  if (deduplicationCache.has(idempotencyKey)) {
    return res.status(200).send(deduplicationCache.get(idempotencyKey));
  }

  res.on('finish', () => {
    if (res.statusCode < 400) {
      deduplicationCache.set(idempotencyKey, res.locals.responseData);
      setTimeout(() => deduplicationCache.delete(idempotencyKey), 60000); // Cache expiration
    }
  });

  next();
}

app.post('/api/resource', deduplicationMiddleware, (req, res) => {
  // Processing the request
  const responseData = { success: true, message: 'Resource created' };
  res.locals.responseData = responseData;
  res.status(201).send(responseData);
});

app.listen(3000, () => console.log('Server running on port 3000'));

Key Concepts

• Idempotency Key: A unique identifier for requests to ensure that the same operation is not performed multiple times.
• Cache: A temporary store to keep track of processed keys and responses.
• Middleware: A function that intercepts and processes requests before reaching the endpoint logic.

By implementing the above deduplication strategy, you can ensure that your API processes each request only once, improving efficiency and resource management.