Retry Without Paying Twice: Idempotency Keys and the Draft That Expired in April

The message arrives at 11:40 on a Thursday night, and it is always some version of the same message: the customer was charged twice.

The trace is not mysterious. A checkout POST went out over a mobile connection in Chattogram. The gateway took twenty-eight seconds. The client’s HTTP timeout was twenty. The client did the sensible thing — it retried — and the second request arrived while the first was still in flight, was authorised on its own merits, and produced a second charge against a card that had already been debited. Two requests. Two successes. One order. Nothing crashed, nothing was misconfigured, and no line of code was wrong in isolation.

What went wrong is architectural, and it has a name. The client could not distinguish “the request never arrived” from “the request arrived, succeeded, and the response was lost.” That distinction is not recoverable from the network. It has to be recovered from the application.

The industry’s answer is a request header called Idempotency-Key. Almost every payment API ships it. Almost every API-design guide recommends it. And there is no RFC that defines it. The Internet-Draft that tried expired on 18 April 2026 without ever reaching the IESG, the field name does not appear in IANA’s HTTP field-name registry, and the two largest implementations disagree about how long the header is honoured by a factor of seven to fourteen.

What we have instead is folklore — a shape copied from Stripe’s documentation and reimplemented from memory a few thousand times. The folklore is right about the easy part and silent on the five decisions that actually determine whether your implementation is correct. This post is about those five decisions.

Part 1: A 1984 paper already told us where this belongs

Before the header, the argument. Saltzer, Reed, and Clark published “End-to-End Arguments in System Design” in ACM Transactions on Computer Systems in November 1984, and its abstract lists — by name — the exact problem in the paragraph above: “duplicate message suppression”¹.

The paper’s central claim is stated once, plainly:

“The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the end points of the communication system. Therefore, providing that questioned function as a feature of the communication system itself is not possible.”¹

Then it applies that claim to precisely our failure:

“reliable detection of a system crash is problematical: the problem may just be a lost or long-delayed acknowledgement. If so, the retried request is now a duplicate, which only the application can discover. Thus the end-to-end argument again: if the application level has to have a duplicate-suppressing mechanism anyway, that mechanism can also suppress any duplicates generated inside the communication network, so the function can be omitted from that lower level.”¹

Only the application can discover it. Not TCP, which will happily deliver a second, well-formed, independently valid request. Not the load balancer. Not your service mesh’s retry policy, which is in fact a duplicate generator. The knowledge that request B is a repetition of request A exists nowhere in the system except in the client’s intent, and the only way to move that knowledge across the wire is for the client to say so explicitly.

An idempotency key is exactly that: the client’s intent, serialised. Everything else in this post is implementation detail around a forty-two-year-old conclusion.

Part 2: What HTTP actually promises, and what it does not

RFC 9110, the June 2022 revision of HTTP’s core semantics, defines idempotency in Section 9.2.2:

“A request method is considered ‘idempotent’ if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request. Of the request methods defined by this specification, PUT, DELETE, and safe request methods are idempotent.”²

Two things follow that are worth stating precisely, because both are routinely misread.

First, idempotency is a property of the intended effect on the server, not of the response. The spec says so directly: “a server is free to log each request separately, retain a revision control history, or implement other non-idempotent side effects for each idempotent request.”² A DELETE that returns 204 and then 404 is still idempotent. If your definition of idempotent is “returns the same bytes,” you are measuring the wrong thing.

Second, and this is the operative sentence for anyone writing a client:

“A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent, regardless of the method, or some means to detect that the original request was never applied.”²

Read the escape clause. HTTP does not forbid retrying a POST. It forbids retrying a POST blindly. It explicitly contemplates a client that has “some means to know that the request semantics are actually idempotent, regardless of the method.” An idempotency key is that means. RFC 9110 leaves the door open and declines to say what walks through it.

The spec also anticipates what happens when people improvise: “Some clients take a riskier approach and attempt to guess when an automatic retry is possible” — retrying a POST, for instance, when the transport connection closed before any part of a response arrived². Every HTTP client library you use in production does some version of this, and most do it by default.

The first attempt, in 2005

Mark Nottingham tried to close the gap twenty-one years ago. POST Once Exactly — draft-nottingham-http-poe-00, published 19 March 2005 — opens with an observation that has aged into a period piece and a permanent truth at the same time:

“A very common sight on the Web is an admonishment to only ‘click submit once.’”³

POE’s mechanism was server-centric rather than client-centric. A server would advertise single-use “POE resources” via a POE-Links response header; the client would POST to one of those URIs, and the resource would guarantee that “a server MUST NOT respond to a POST with a 2xx-series status code if it has been successfully POSTed to at some point in the past,” answering any repeat with 405 Method Not Allowed³. A 2xx meant you did it; a 405 meant you already did it. Either way the client learned the truth.

The draft expired on 17 September 2005 and was never revised. The idea it was reaching for — that the server should hand out the deduplication token — was quietly correct, and we will come back to it in Part 11.

Part 3: The draft everybody cites expired in April

The modern attempt is draft-ietf-httpapi-idempotency-key-header, by Jayadeba Jena and Sanjay Dalal. It is a genuine IETF working-group document — adopted by the HTTPAPI WG, marked Standards Track — and revision 07 was posted on 15 October 2025⁴.

It is also dead. The IETF Datatracker records its document state as Expired, with an IESG state of “I-D Exists” — described in the tracker as “The IESG has not started processing this draft, or has stopped processing it without publication.” The expiry timestamp is 2026-04-18T13:37:07Z⁵. The working group itself is in good health — five of its drafts are active and six have been published as RFCs — but three have expired, and this is one of them. The individual submission it grew from, draft-idempotency-header, is marked “Replaced” by this document. Nothing is marked as replacing this one⁵.

Three further details are worth putting on the record, because they tell you how much weight the document can bear.

It is not registered. Section 3.1 asks IANA to add Idempotency-Key to the Hypertext Transfer Protocol (HTTP) Field Name Registry with status “permanent”⁴. As of this writing the registry lists no such field⁶. The header your payment API ships is, formally, an unregistered non-standard field.

Its normative references have themselves been obsoleted. The draft normatively cites RFC 8941 for Structured Fields, RFC 7807 for problem details, and RFC 4122 for UUIDs⁴. RFC 8941 was obsoleted by RFC 9651 in September 2024⁷. RFC 4122 was obsoleted by RFC 9562 in May 2024⁸. RFC 7807 was obsoleted by RFC 9457 in July 2023 — and RFC 9457’s third listed author is Sanjay Dalal⁹, the same co-author. Revision 07 of the draft was posted in October 2025, twenty-seven months after that obsoletion, and still cites RFC 7807.

Nobody sends the header the way the draft defines it. Section 2.1 is unambiguous: “Idempotency-Key is an Item Structured Header. Its value MUST be a String,” and the draft’s examples are correspondingly quoted: Idempotency-Key: "8e03978e-40d5-43e8-bc93-6894a57f9324"⁴. Structured Fields requires it: “When serialized in a textual HTTP field, Strings are delimited with double quotes”⁷. Stripe’s documentation shows -H "Idempotency-Key: KG5LxwFBepaKHyUD" — no inner quotes¹⁰. Adyen documents idempotency-key:<key> — likewise¹¹.

The failure mode is not symmetric, either. A bare token like KG5LxwFBepaKHyUD parses under Structured Fields as a Token, not a String — the wrong type, silently. But an unquoted UUID is usually not even that: sf-token = ( ALPHA / "*" ) *( tchar / ":" / "/" ) requires the first character to be a letter⁷, and the first character of a UUIDv4 is a uniformly random hex digit. Ten of the sixteen possible values are digits. Feed an unquoted UUIDv4 to a conforming Structured Fields parser and it fails outright about five times in eight, and parses as the wrong type the other three.

The practical consequence: you cannot implement this header with a Structured Fields parser and remain compatible with the ecosystem. Everyone reads the raw field value as an opaque string, and that is the right call. It is worth knowing that it is a deliberate deviation from the only specification that exists.

Part 4: Decision one — what the key is scoped to

Here is where the folklore runs out. The draft says the key “MUST be unique and MUST NOT be reused with another request with a different request payload,” and then hands the hard part back: “Uniqueness of the key MUST be defined by the resource owner and MUST be implemented by the clients of the resource.”⁴

That is the whole specification of scope. The resource owner decides. So decide deliberately, because the naïve choice is a vulnerability.

If your lookup key is the idempotency key alone, then the key is a globally addressable handle to a stored response. The draft’s own security considerations spell out what follows: “When an idempotency implementation allows low entropy keys, attackers MAY determine other keys and use them to fetch existing idempotent cache entries, belonging to other clients.”⁴ A tenant who guesses or brute-forces a short key reads another tenant’s payment confirmation — response body, amount, customer reference and all. The draft’s recommended mitigation is the right one: “implement a unique composite key as the idempotent cache lookup key. For example, a composite key MAY be implemented by combining the idempotency key sent by the client with other client specific attributes known only to the resource.”⁴

In practice the lookup key should be at minimum (tenant_id, endpoint, idempotency_key). Including the endpoint matters more than it looks: without it, a client that reuses one key across POST /payments and POST /refunds gets a refund’s stored response replayed for a payment, or a fingerprint mismatch that reads as a client bug when it is really a scoping bug.

AWS is the instructive example of scope taken seriously. EC2’s client token is “a unique, case-sensitive string of up to 64 ASCII characters,” and idempotency is explicitly either Regional or Zonal: “if you specify the same client token in two calls to AllocateHosts in the same Region, the calls succeed if they specify different values for the AvailabilityZone parameter.”¹² RunInstances picks between the two depending on whether you named an Availability Zone. Reuse the same token in us-east-1 and eu-west-1 and you get two instances, by design.

That is a scope decision with real consequences, published, in the API reference. Yours should be too — and the draft agrees on the obligation if not the content: “Resources MUST publish a idempotency related specification.”⁴

Part 5: Decision two — how long the key lives

The draft’s position is that this is your problem: “The resource MAY require time based idempotency keys to be able to purge or delete a key upon its expiry. The resource SHOULD define such expiration policy and publish it in the documentation.”⁴

The two largest published policies do not agree.

Stripe: “You can remove keys from the system automatically after they’re at least 24 hours old. We generate a new request if a key is reused after the original is pruned.”¹⁰

Adyen: “Idempotency keys are valid for a period of 7 to 14 days after first submission.”¹¹

AWS EC2: the idempotency page publishes no retention window at all. What it publishes instead is the mismatch behaviour — a retry with the same client token and different parameters fails with IdempotentParameterMismatch¹² — which is a different, and arguably safer, way to be unambiguous.

Two of these are payment APIs, and their windows differ by a factor of seven to fourteen. The divergence is not cosmetic — it changes the failure mode at the boundary. Read Stripe’s sentence again, carefully: we generate a new request if a key is reused after the original is pruned. The protection does not degrade into an error. It degrades into silence. A retry that arrives 25 hours late is not rejected; it is executed. If your client has a dead-letter queue that drains on a daily cron, or an operator who reruns a failed batch on Monday morning, you have built a machine for generating exactly the duplicate charge the key was meant to prevent.

The rule that falls out is narrow and worth stating explicitly: a client’s maximum retry horizon must be strictly shorter than the server’s minimum key retention. If you cannot guarantee that — and with human-triggered replays you cannot — the idempotency key is not sufficient on its own. You need a second, permanent uniqueness constraint in your own domain: an order reference, an invoice number, something with business meaning and a unique index on it that outlives any cache. The idempotency key handles the thirty-second retry. The domain constraint handles the thirty-day one.

Adyen also caps its key at 64 characters, Stripe at 255¹¹¹⁰. If you are writing a client that talks to both, your key generator is bounded by the smaller. A UUIDv4 in canonical form is 36 characters and clears both, which is one more reason the draft’s recommendation to use one is good advice.

Part 6: Decision three — the fingerprint, and the 422 it produces

Suppose the key arrives twice with two different bodies. The draft calls the discriminator an idempotency fingerprint: “generated from request payload data by the resource,” with several permitted strategies — “Checksum of the entire request payload,” “Checksum of selected element(s) in the request payload,” “Field value match for selected element(s) in the request payload,” and others⁴.

Note the word MAY. Fingerprinting is optional in the draft. It should not be optional in your implementation, because without it a client bug that reuses one key for two genuinely different payments silently returns the first payment’s response for the second — a success the caller believes and a charge that never happened.

The prescribed response when the fingerprint differs is 422 Unprocessable Content, with an RFC 7807 problem document:

HTTP/1.1 422 Unprocessable Content
Content-Type: application/problem+json
Content-Language: en
{
  "type": "https://developer.example.com/idempotency",
  "title": "Idempotency-Key is already used",
  "detail": "This operation is idempotent and it requires
   correct usage of Idempotency Key. Idempotency Key MUST not be
   reused across different payloads of this operation.",
}

Two notes on that block, both from the primary source. The trailing comma after "detail" is in the draft as published — it is not valid JSON⁴. And Content-Type: application/problem+json should now cite RFC 9457, not the obsoleted RFC 7807; the media type is unchanged, the reference is not⁹.

The design question the draft does not answer is what the fingerprint covers, and it is the one that will bite you. Hash the raw request bytes and the fingerprint is correct but brittle: a client that re-serialises its JSON with different key ordering, or upgrades a library that changes float formatting, gets a 422 on a retry that is semantically identical. Hash a canonical projection of the significant fields — amount, currency, payee, reference — and it survives serialisation drift, at the cost of deciding per endpoint which fields count.

Choose the second, and write the projection down next to the endpoint’s schema rather than deriving it dynamically. A fingerprint that silently changes when someone adds an optional field to a request DTO is a 422 waiting to happen in production, six weeks after the field was added, to the one client that retries.

AWS’s equivalent is worth noting because it draws the line differently again: a retry “using the same client token, but one or more of the parameters are different, other than the Region or Availability Zone” fails with IdempotentParameterMismatch¹² — full parameter comparison, with two named exemptions that exist precisely because they are the scope dimensions from Part 4.

Part 7: Decision four — the 409 is a promise, not an error

The concurrent case is the one from the opening paragraph, and it is the case most implementations get wrong.

The draft’s rule: “The request was retried before the original request completed. The resource SHOULD respond with a resource conflict error” — 409 Conflict, titled “A request is outstanding for this Idempotency-Key”⁴. Adyen returns “HTTP 422 – Unprocessable Entity or HTTP 409 - Conflict status with the error code 704: ‘request already processed or in progress’”¹¹. Stripe declines to store a result at all in this case: “If incoming parameters fail validation, or the request conflicts with another request that’s executing concurrently, we don’t save the idempotent result because no API endpoint initiates the execution. You can retry these requests.”¹⁰

The important thing about the 409 is what it means to the client, and it is the opposite of what “Conflict” suggests. A 409 here is not a failure. It is the server saying: your request is being processed right now, and I am refusing to process it a second time. It is the single most valuable response in the whole protocol, because it is the only one that converts the client’s unresolvable ambiguity — did it arrive? — into a definite answer: yes, it arrived, wait.

A client that treats 409 as terminal and surfaces “payment failed” to the user has taken a successful deduplication and turned it into a support ticket. The correct client behaviour is to back off and retry the same key until it gets a stored response.

Making that promise true requires the in-flight record to be written before any side effect, in its own committed transaction. The sequence is:

1. INSERT the key row with state in_progress, and commit. If the insert conflicts, this is a duplicate — go to step 4.
2. Perform the work.
3. Update the row to completed with the stored status and body, and commit.
4. On conflict: if completed, replay. If in_progress and the lease is live, 409. If in_progress and the lease has expired, the original worker died — recover.

PostgreSQL gives you step 1 for free with a unique index and ON CONFLICT DO NOTHING. What it does not give you for free is step 4’s last branch, and skipping it is the classic production incident: a worker is OOM-killed between steps 1 and 3, the key row is stranded in in_progress forever, and every subsequent retry of that key gets a 409 for eternity. The key becomes a tombstone on a payment that never happened. Hence the lease: locked_at, a bounded lifetime, and an explicit takeover path.

It is worth knowing the error codes you will actually see. PostgreSQL signals a unique-index conflict as 23505 unique_violation and a serialisation conflict under SERIALIZABLE as 40001 serialization_failure¹³. Both are retryable at the transaction level and neither should be allowed to escape as a 500. And there is a third code in Class 40 that deserves to be better known, because it is this entire post compressed into one identifier: 40003 statement_completion_unknown¹³. Your database has an error code for I cannot tell you whether that worked. So does the network. The idempotency key exists because that state is unavoidable and someone has to resolve it.

Part 8: Decision five — where the replayed response comes from

The last decision is the one the draft does not address at all, and it is where most real implementations quietly fail.

Steps 1–3 above assume the work is a single database transaction. Payments are not. A payment handler debits a ledger, calls a gateway over the network, writes a gateway reference, enqueues a receipt email. Brandur named the structure in October 2017 and the vocabulary has held up: a foreign state mutation is “calling out and manipulating data on another system. This might be creating a charge on Stripe, adding a DNS record, or sending an email.” An atomic phase is “a set of local state mutations that occur in transactions between foreign state mutations.” A recovery point is “a name of a check point that we get to after having successfully executed any atomic phase or foreign state mutation.”¹⁴

The reason this matters is that you cannot wrap a gateway call in a database transaction. Once the charge exists at the gateway, no ROLLBACK of yours will remove it. So the naïve implementation — begin transaction, do everything, store the response, commit — is not merely inelegant. It is wrong in a way that costs money: crash after the gateway call and before the commit, and the key row rolls back to in_progress or vanishes, while the charge stands. The retry finds no completed record and charges again.

The fix is to make the recovery point advance with each atomic phase, in that phase’s own transaction, so that a retry resumes from the last committed checkpoint rather than restarting. A handler that has completed the gateway call and crashed before enqueuing the receipt must, on retry, skip the gateway call and enqueue the receipt. That is only possible if the checkpoint was committed, and only correct if the gateway call itself was idempotent — which is why you pass your own idempotency key downstream to the gateway as well. The property has to hold at every hop or it holds at none.

The corollary for the stored response: persist the status code and body as part of the transaction that reaches the terminal recovery point, never afterwards as a separate write. Stripe’s behaviour here is the one to copy, and it is stricter than most people implement — Stripe saves “the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors.”¹⁰

That last clause is counterintuitive and correct. Replaying a 500 the server genuinely produced is honest; re-executing instead would send the retry down a different code path than the first attempt, which is precisely the non-determinism idempotency exists to eliminate.

Part 9: A reference implementation

What follows is the shape, not a library. It targets PostgreSQL and is written to be read; the transaction boundaries are the part that matters, and they are the part most often collapsed.

CREATE TABLE idempotency_keys (
    tenant_id      uuid        NOT NULL,
    endpoint       text        NOT NULL,
    key            text        NOT NULL,
    fingerprint    bytea       NOT NULL,
    state          text        NOT NULL CHECK (state IN ('in_progress','completed')),
    recovery_point text        NOT NULL DEFAULT 'started',
    response_code  int,
    response_body  jsonb,
    locked_at      timestamptz,
    created_at     timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (tenant_id, endpoint, key)
);

-- Retention is a published contract, not a cleanup job that happens to run.
CREATE INDEX idempotency_keys_created_at_idx ON idempotency_keys (created_at);

The primary key is the scope decision from Part 4, expressed where the database can enforce it. The recovery_point column is Part 8. The locked_at column is the lease from Part 7.

import hashlib
import json
from datetime import datetime, timedelta, timezone

LEASE = timedelta(seconds=90)


def fingerprint(payload: dict, significant: tuple[str, ...]) -> bytes:
    """Hash a canonical projection of the semantically significant fields.

    Deliberately not a hash of the raw bytes: a client that re-serialises
    an identical request must not receive a 422 (Part 6).
    """
    projection = {k: payload[k] for k in significant if k in payload}
    canonical = json.dumps(projection, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).digest()


def claim(cur, tenant, endpoint, key, fp):
    """Claim the key, or report why we cannot. The caller commits this alone."""
    cur.execute(
        """
        INSERT INTO idempotency_keys
            (tenant_id, endpoint, key, fingerprint, state, locked_at)
        VALUES (%s, %s, %s, %s, 'in_progress', now())
        ON CONFLICT (tenant_id, endpoint, key) DO UPDATE
            SET locked_at = now()
            WHERE idempotency_keys.state = 'in_progress'
              AND idempotency_keys.locked_at < now() - %s
              AND idempotency_keys.fingerprint = EXCLUDED.fingerprint
        RETURNING recovery_point
        """,
        (tenant, endpoint, key, fp, LEASE),
    )
    row = cur.fetchone()
    if row is not None:
        # Fresh insert, or a dead lease reclaimed under a matching fingerprint.
        return ("owned", row[0])

    # RETURNING yielded nothing: the row exists and is not ours to take.
    cur.execute(
        """
        SELECT state, fingerprint, response_code, response_body
          FROM idempotency_keys
         WHERE tenant_id = %s AND endpoint = %s AND key = %s
        """,
        (tenant, endpoint, key),
    )
    state, stored_fp, code, body = cur.fetchone()
    if stored_fp != fp:
        return ("mismatch", None)
    if state == "completed":
        return ("replay", (code, body))
    return ("in_flight", None)

The dispatch that wraps it is four branches and no cleverness:

def handle(conn, tenant, endpoint, key, payload, significant, do_work):
    if key is None:
        return problem(400, "Idempotency-Key is missing")

    fp = fingerprint(payload, significant)

    with conn:                                   # transaction 1: claim only
        with conn.cursor() as cur:
            outcome, extra = claim(cur, tenant, endpoint, key, fp)

    if outcome == "mismatch":
        return problem(422, "Idempotency-Key is already used")
    if outcome == "in_flight":
        return problem(409, "A request is outstanding for this Idempotency-Key")
    if outcome == "replay":
        code, body = extra
        return (code, body)

    # outcome == "owned": we hold the lease. Resume from `extra`, the last
    # committed recovery point — each phase commits its own advance (Part 8).
    code, body = do_work(conn, resume_from=extra)

    with conn:                                   # transaction N: terminal state
        with conn.cursor() as cur:
            cur.execute(
                """
                UPDATE idempotency_keys
                   SET state = 'completed', recovery_point = 'finished',
                       response_code = %s, response_body = %s::jsonb, locked_at = NULL
                 WHERE tenant_id = %s AND endpoint = %s AND key = %s
                """,
                (code, json.dumps(body), tenant, endpoint, key),
            )
    return (code, body)

Three properties are load-bearing, and each corresponds to a failure I have watched happen.

The claim commits in its own transaction before do_work runs. If it shared a transaction with the work, a crash would roll the claim back and the retry would re-execute.

The ON CONFLICT ... DO UPDATE ... WHERE clause is the lease takeover. A row whose worker died more than ninety seconds ago is reclaimable; a live one is not. Without it, one OOM kill strands a key permanently in 409.

The fingerprint guards the takeover too — it is a condition of the DO UPDATE, not an afterthought in the application. A stranded key reclaimed by a request carrying a different payload must produce a 422, not a silent execution under someone else’s key, and it must not have its lease stolen on the way out.

The lease duration is a real trade-off and there is no correct universal value. Too short and two workers process concurrently; too long and a genuine crash blocks retries for the whole window. Set it above your endpoint’s p99.9 latency and treat a takeover as an event worth logging, because a takeover means something died.

Part 10: The same trick, everywhere else

It is clarifying to notice that every system claiming “exactly once” is doing this, and that the differences are mostly about who generates the key.

Kafka’s idempotent producer, from KIP-98, moves the identity to the broker: “Every new producer will be assigned a unique PID during initialization,” and “sequence numbers will start from zero and be monotonically increasing, with one sequence number per topic partition produced to.” The broker “will reject a produce request if its sequence number is not exactly one greater than the last committed message from that PID/TopicPartition pair. Messages with a lower sequence number result in a duplicate error, which can be ignored by the producer.”¹⁵

That design buys something HTTP keys do not: the producer never has to remember anything durable, because the broker holds the high-water mark. It also pays a price the KIP states plainly: “since each new instance of a producer is assigned a new, unique, PID, we can only guarantee idempotent production within a single producer session.”¹⁵ Restart the producer and the guarantee resets. Kafka’s “exactly once” is exactly-once within a session; crossing that boundary needs transactions.

Google Cloud Pub/Sub attacks the same problem from the subscriber side, by making acknowledgement identities single-use. After a valid redelivery, “only the latest acknowledgment ID can be used to acknowledge the message. Any requests with a previous acknowledgment ID fail.”¹⁶ The guarantee is also narrower than the feature name suggests: “Only the pull subscription type supports exactly-once delivery, including subscribers that use the StreamingPull API. Push and export subscriptions don’t support exactly-once delivery.”¹⁶ Push delivery — the mode that looks most like an HTTP webhook — does not get it, which is not a coincidence.

The pattern is the same one every time, and my read of it is that “exactly once” is always a description of the outcome, never of the delivery. What the delivery gives you is at-least-once; what makes the outcome exactly-once is a durable record of what has already been applied. The only real questions are who mints the identifier, where the record lives, and how long it is kept. HTTP’s answer — client mints, server stores, retention undefined — is the loosest of the three, which is exactly why the five decisions in this post are yours to make.

Part 11: When you do not own the client

Everything so far assumes the caller can be told to send a key. For inbound webhooks and payment notifications, it cannot.

SSLCOMMERZ, one of Bangladesh’s principal payment gateways, documents its Instant Payment Notification like this: “For any notification, SSLCOMMERZ will send HTTP message in POST method called IPN Message to the Listener which is to be configured by the Merchant at their SSLCOMMERZ Administrator Panel.” The merchant is then instructed to call a separate validation endpoint before believing anything, and the documentation is explicit that both val_id and tran_id “need to be validated with your system database”¹⁷.

What the documentation does not specify — and I checked, because the temptation to assert it is strong — is a retry policy or a duplicate-delivery guarantee. That absence is the design input. When a sender publishes no delivery semantics, you must assume at-least-once, because at-least-once is what every retrying HTTP sender does whether or not it says so.

Since the sender will not give you a key, derive one. This is where POE’s 2005 instinct was right after all: the deduplication token should come from whoever owns the identity of the operation. For an IPN, that is the transaction identifier the gateway assigned. Two rules make it safe:

Store the dedup record in the same transaction as the effect. If handling an IPN marks an order paid, then INSERT INTO processed_notifications (source, tran_id) and UPDATE orders SET status='paid' must be the same transaction. A 23505 unique_violation¹³ on the insert is the duplicate detection — no read-then-write race, no advisory lock, no window. Return 200, do nothing else.

Never derive the key from the payload alone. Hashing the notification body seems attractive and fails on the case that matters: a gateway that legitimately sends two distinct notifications with identical bodies — a retry of a webhook whose only distinguishing field is a delivery ID you discarded — becomes indistinguishable from a duplicate. Key on the sender’s operation identifier, and keep the delivery identifier separately for your audit log.

The same applies to webhooks you send. Publish your retry schedule and put a stable event ID in every delivery, so your consumers can build that guard without guessing. Leaving it out is what pushes every downstream integrator into defensive folklore — and you have read this far, so you know what that costs.

Part 12: What to actually do

The five decisions, as a checklist:

1. Scope. Make the lookup key (tenant, endpoint, idempotency_key) at minimum, and enforce it as a primary key, not in application code.
2. Retention. Publish the window — then add a permanent domain-level uniqueness constraint anyway, because Stripe’s documented behaviour after pruning is to execute, not to reject¹⁰, and no retention window is longer than an operator’s memory.
3. Fingerprint. Hash a canonical projection of the significant fields, not the raw bytes, and write the projection down beside the schema. Return 422 on mismatch, with an RFC 9457 problem document⁹.
4. Concurrency. Commit the in_progress row before any side effect, hold a bounded lease, and return 409 while it is live. Document that 409 as retry the same key.
5. Recovery. Advance a committed recovery point after each atomic phase, pass your key downstream to every foreign system that accepts one, and store the response — errors included — in the transaction that reaches the terminal state.

Two things to write down in your API reference, since no RFC will: the retention window, and whether a mismatched repeat returns 422 or 409. Those are the facts an integrator cannot discover by experiment without risking a duplicate charge in production.

And one thing not to do: do not send the header quoted as the draft requires unless you have confirmed your counterparty parses Structured Fields. The specification and the ecosystem disagree, the ecosystem won, and a "-wrapped key will be treated as a different key by every implementation that compares raw field values.

Closing

There is a temptation to read Part 3 as an indictment — a working group that let a useful draft lapse, an IANA registration that never happened, three normative references quietly rotting. It is worth resisting. The IETF standardises interoperable wire formats, and Idempotency-Key is not really a wire format problem. The header itself is one sentence of specification. Everything that determines whether an implementation is correct — scope, retention, fingerprint projection, lease duration, recovery-point placement — is a property of the resource’s own semantics, and the draft was honest enough to say so, in as many words: uniqueness MUST be defined by the resource owner.

What that leaves is a genuinely unusual situation. This is one of the few load-bearing pieces of modern API design where there is no specification to be conformant with, only a body of practice — and the practice, when you read the primary sources side by side rather than the blog posts about them, disagrees with itself about retention by a factor of seven to fourteen, about status codes, about key length, and about whether the header is even syntactically a string.

So the burden is yours, and it is not the burden of writing the middleware. Part 9 is a hundred lines and most teams have already written it. The burden is publishing what you chose. An idempotency key is a contract between a client and a server about a question neither can answer alone — did it happen? — and a contract that only one party has read is not a contract. It is folklore with a header name.

Write down your five answers. Put them in the API reference, next to the endpoint. That is the part nobody does, and it is the only part the client cannot discover without charging someone twice.

References