DynamoDB Idempotency Keys: The Right Way to Deduplicate Writes
Your distributed system will deliver some messages twice. A client retries a request that actually succeeded but timed out. A webhook fires twice because the consumer didn’t respond fast enough. A Lambda triggered by SQS gets reinvoked. A user double-clicks “Submit.”
The fix is an idempotency key - a unique value the client provides with the request, used to detect “I’ve seen this before” and short-circuit the duplicate. Stripe popularized this pattern for payments; it works for any write that has visible side effects.
In DynamoDB, idempotency keys are cheap and clean to implement, but there are several wrong ways and one or two right ones. This post is the playbook.
The shape of the pattern
For any write that needs to be idempotent (creating an order, charging a card, sending an email):
- The client generates an idempotency key (UUID, ULID, or a deterministic hash of the request payload).
- The server stores the key plus the result of the operation.
- On retry with the same key, the server detects the duplicate and returns the original result.
Two states matter: “haven’t seen this key” and “already processed this key.” Idempotency is just persistent memory of which is which.
DynamoDB’s attribute_not_exists conditional write is the primitive that makes this trivial.
The minimal implementation
async function createOrder(idempotencyKey: string, order: OrderInput) {
try {
await IdempotencyEntity.create({
idempotencyKey,
operation: "createOrder",
ttl: nowSeconds() + 24 * 60 * 60, // 24h retention
result: { status: "pending" },
})
.where(({ idempotencyKey: k }, { notExists }) => notExists(k))
.go()
} catch (err) {
if (err.name === "ConditionalCheckFailedException") {
// Already processed - fetch and return original result
const existing = await IdempotencyEntity.get({ idempotencyKey }).go()
return existing.data.result
}
throw err
}
// First time - actually do the work
const result = await actuallyCreateOrder(order)
// Save the result so retries return it
await IdempotencyEntity.update({ idempotencyKey })
.set({ result, status: "completed" })
.go()
return result
}This works for most cases. There’s a window between the idempotency record being created and the result being stored where a retry would see a half-finished record - we’ll fix that in a moment.
The transactional version (preferred)
The cleanest implementation puts the idempotency record AND the actual write in a single transaction. If the idempotency key is taken, the transaction fails and nothing is written. If it succeeds, both the idempotency record and the resource are created atomically.
async function createOrder(idempotencyKey: string, order: OrderInput) {
const orderId = ulid()
try {
await client.send(new TransactWriteItemsCommand({
TransactItems: [
{
Put: {
TableName,
Item: marshall({
pk: `IDEMPOTENCY#${idempotencyKey}`,
sk: "#METADATA",
orderId,
ttl: nowSeconds() + 24 * 60 * 60,
}),
ConditionExpression: "attribute_not_exists(pk)",
},
},
{
Put: {
TableName,
Item: marshall({
pk: `ORDER#${orderId}`,
sk: "#METADATA",
...order,
}),
},
},
],
}))
return { orderId }
} catch (err) {
if (err.name === "TransactionCanceledException" &&
err.CancellationReasons[0]?.Code === "ConditionalCheckFailed") {
// Duplicate - look up the original orderId
const existing = await IdempotencyEntity.get({ idempotencyKey }).go()
return { orderId: existing.data.orderId }
}
throw err
}
}This is the cleanest version because:
- Idempotency check and order creation are atomic.
- No half-finished state - either both writes happen or neither does.
- Retries return the original
orderIddeterministically.
The cost is 2x WCU per write (transactions cost double, see transactions vs conditional writes). For high-volume idempotent operations, that matters. Worth it for anything money-related.
What goes in the idempotency record
At minimum: the key itself. Practically, you’ll also want:
operation- which kind of operation this key was used for. Prevents the same key being reused for different operation types.ttl- how long the key is remembered. Stripe uses 24 hours; for less critical operations, 1 hour is plenty.resultorresourceId- what to return on retry. For “create order,” the orderId is enough; the caller can fetch the order separately.status- “pending” / “completed” / “failed”. Lets callers distinguish “still processing” from “finished.”requestHash(optional) - hash of the request payload. Lets you detect “client reused the same idempotency key with a different request” - which should be an error.
The requestHash is the subtle one. Stripe explicitly checks this: if you reuse an idempotency key with a different request, they reject it. Without that check, a client bug could cause “create $5 charge” to silently return the result of a previous “$50 charge” that used the same key.
Picking the key
Two common approaches:
Client-generated keys: the client supplies a UUID per logical operation. Most flexible. Works for “user clicked submit twice” because the same UUID is included in both attempts.
Deterministic hashes: the server computes a hash of the request payload (including a per-user nonce or something to distinguish similar requests). No client coordination needed, but harder to handle edge cases like “create two identical orders deliberately.”
Stripe-style APIs use client-generated keys via an Idempotency-Key header. Lambda-driven systems often use a hash of the SQS message ID or DynamoDB Stream sequence number, since the framework already provides a unique ID per message.
TTL on the idempotency record
Idempotency records don’t need to live forever. The retry window for any given operation is bounded - after some time, you can safely forget the key.
- Synchronous APIs - 1 hour is usually enough. Network retries happen within seconds; user retries within minutes.
- Webhooks / event-driven - 24 hours, or whatever the upstream service’s redelivery window is. Stripe redelivers webhooks for up to 3 days.
- Critical financial operations - 7-30 days, conservatively.
Use DynamoDB TTL to clean these up automatically. Don’t pay for storing dead idempotency keys forever.
Handling failures
If the operation fails (the order can’t be created because of a downstream error), what should the idempotency record contain?
There are two approaches. You can store the failure: update the idempotency record with status: "failed" and the error, so retries return that error. This is deterministic and stops retries from hammering a broken downstream, but a user who fixes their request and retries with the same key gets the old error.
Or you can delete the record on failure, letting a retry try fresh. More forgiving, but might cause cascading retries if the failure is upstream.
Stripe stores failures and exposes them on retry. For payment systems, this is the right call - “your card was declined” should be deterministic, not a roll of the dice. For most non-financial operations, deleting the key on failure is fine.
If you delete on failure, do it carefully: only for transient errors you’re confident the retry will fix. Don’t delete the key on validation errors (the client just sent bad data and the retry will fail the same way).
The “in-flight” problem
What if a duplicate request arrives while the first request is still being processed?
Three options. Block the duplicate: return a 409 Conflict immediately. The client can retry later. Simplest. Wait for the first to finish: poll the idempotency record until it’s complete. Adds latency to the duplicate request but the caller gets a consistent result. Or treat as separate requests: don’t use idempotency at all, process both. Loses the dedupe property - bad idea unless your operation is naturally idempotent.
For most APIs, blocking is the right call. The client retries are usually rare enough that 409 → backoff → success is fine. Add “in_progress” status to the idempotency record so callers can distinguish.
// On retry while first request is still processing
if (existing.data.status === "in_progress") {
throw new Conflict("operation in progress, retry shortly")
}Common bugs
Reusing the same key for different operation types: without an operation check, a “create order” idempotency key can accidentally short-circuit a “cancel order” with the same key. Always namespace the key by operation type or include the operation in the record.
No TTL: idempotency records pile up forever, costing storage. Always set a TTL.
Returning stale results: if the original operation’s result is a snapshot (e.g. “order with status ‘pending’”), the retry returns that snapshot even if the order has since shipped. Return only stable identifiers (orderId) and let the caller fetch the current state.
Storing the entire response in the idempotency record: big responses balloon the storage cost and require keeping the same response shape across deploys. Store the resource ID and let the caller re-fetch.
Forgetting the request hash: without it, clients can mix up keys and cause silent data corruption. Include a hash; reject mismatches.
Treating attribute_not_exists as a single point in time: it is - but if you split idempotency check and resource creation into separate writes (non-transactional), you have a race window. Use a transaction or accept the tiny race.
When idempotency keys are unnecessary
Some operations are naturally idempotent and don’t need explicit keys:
- Create-or-update by deterministic ID - if the ID is derived from the request (e.g.
userId-dayfor a daily report), the second write just overwrites the first. PUTsemantics - replacing an item by ID. Repeating the samePutproduces the same end state.- Counters with
ADD- depending on whether duplicates should add twice (they will - so this is not idempotent for counters).
Be careful about claims of “naturally idempotent.” Actually trace what happens if the operation runs twice. Most “create” operations, despite the friendly name, are not idempotent without help.
A note on at-most-once delivery
People sometimes ask “can DynamoDB give me exactly-once writes?” Strictly, no - the network can always retry. But idempotency keys + DynamoDB give you the next best thing: at-most-once visible effects, even with at-least-once delivery.
That’s the right mental model. The wire is at-least-once. The application becomes effectively-once via idempotency. DynamoDB is the durable place where the deduplication state lives.
Mental model
An idempotency key is a primary key on the operation, not the resource. The resource has its own ID; the key is the dedup token for “did we already process this request.” Your idempotency table (or item type, in single-table design) is just a registry of keys → results, with TTL.
Once you have one, every “is this a duplicate?” question becomes a GetItem or a conditional PutItem. No race conditions, no scheduled cleanup, no exotic infrastructure. DynamoDB handles the hard parts.
Idempotency keys appear in several patterns at singletable.dev - the e-commerce orders, marketplace, and event sourcing patterns all assume them for write deduplication. Building singletable.dev means making these guarantees visible at design time.
