Exactly-Once Is a Myth: Idempotent Consumers in Practice

On July 18, 2013, Twilio’s billing system tried to recharge some customers’ credit cards, then tried again, and again. A Redis failure had left every balance at zero and unwritable. It hit 1.4% of customers, and some had their accounts suspended.

Your consumer can do the same on any ordinary deploy. Event evt-2 says: credit account 1 with 100. Postgres commits, and the process dies before the Kafka offset commit. On restart it credits the account again, with Kafka’s exactly-once settings on the whole time. The idempotent consumer pattern is how you stop that.

Updated 4 November 2026 · Tested with Python 3.12, confluent-kafka 2.15.1, Kafka 4.3.1, PostgreSQL 18.6

Part 4 of the Reliable Python services series · ← Transactional outbox

The idempotent consumer pattern makes a message handler safe to run more than once. It records each message’s ID in the same database transaction as the side effect, so a redelivered message is skipped.

My position: exactly-once delivery doesn’t exist across a network, but exactly-once effects do, and they’re your consumer’s job. Kafka didn’t lie to you. Its guarantee just ends where your code begins.

Part 3’s transactional outbox pattern in Python left order 101’s email going out, maybe twice. This post makes “twice” harmless. By the end you’ll have:

• The four places duplicates come from, and the default setting that makes them worse.
• A Python consumer that commits the message ID and the side effect in one Postgres transaction.
• What auto-commit, commit-after-the-database and the inbox did across 100 hard kills.
• A retention rule for the inbox, and a plan for side effects outside your database.

We’ll follow evt-2 the whole way. Guess before each answer.

Jargon alert

• At-least-once delivery: the broker redelivers until you acknowledge, so you may see a message twice.
• Offset commit: how a Kafka consumer acknowledges. A restarted consumer starts from the last committed offset.
• Rebalance: the group moving partitions when a consumer joins, leaves or gets thrown out.
• Zombie: a consumer the group has given up on that keeps running, and writing.

Is exactly-once delivery actually possible?

No, not to anything outside the system doing the delivering. An acknowledgement travels over the same unreliable network as the message it confirms. When an ack goes missing, the sender can’t tell a lost message from a lost ack. So it resends and risks a duplicate, or doesn’t and risks a loss. There’s no third option.

It’s the two generals problem. Two generals on opposite hills must agree when to attack, and any messenger crossing the valley can be caught. A’s “dawn” needs B’s reply, which needs a reply of its own, forever. Your broker and consumer are the generals. evt-2’s offset commit was a messenger that never set out, so Kafka sent evt-2 again.

So who’s right: Tyler Treat’s You Cannot Have Exactly-Once Delivery (2015), or Confluent’s Exactly-Once Semantics Are Possible (2017)? Both. Treat means delivery across a network. Confluent means processing inside Kafka, and its post admits a Streams app calling a remote store gets no such guarantee for the side effect.

What you can build is at-least-once delivery plus dedup in the consumer. Microsoft’s Idempotent Consumer pattern page calls that effectively-once processing. It’s the honest name.

Exactly once vs at least once vs at most once: which one do you have?

Almost certainly at-least-once, plus some losses if you use auto-commit carelessly. Every mainstream broker defaults to it, because losing a message is usually worse than seeing it twice.

Spot the pattern? Every built-in dedup works on the send side, inside a window. None stops your consumer from applying evt-2 twice.

What Kafka’s exactly-once covers, and where it stops

It covers writes into Kafka. The idempotent producer stops its own retries from duplicating records, and transactions make a read-process-write loop between topics atomic. The Postgres write, the email and the payment call your consumer makes are outside it.

Think of an office mail room. Inside the building, every memo is logged and lands once. When a courier carries one out the front door, the log stays behind.

Even the producer half has fine print. In the Java client, enable.idempotence has defaulted to true since Kafka 3.0, unless a setting such as acks=1 conflicts (see the producer configs). In Python, librdkafka defaults it to false. And the javadoc says it only covers one producer session.

Transactions plus isolation.level=read_committed give exactly-once between topics, which is what Kafka Streams uses. For other destinations, the Kafka 4.3 design docs say it “generally requires cooperation” from them. Postgres has never heard of your Kafka transaction.

What about KIP-939, a two-phase commit between Kafka and a database? It’s accepted, but the producer has no prepareTransaction() yet and the pull request is still open. Watch it; don’t build on it.

Where do duplicate messages come from?

From four places: a crash between the side effect and the offset commit, a rebalance, a producer or relay that publishes twice, and a lock or visibility timeout that runs out mid-handler. The first two are consumer-group problems. The last two hit any broker.

A crash between the side effect and the offset commit

This is evt-2’s crash. Commit after the side effect, and a crash between the two replays the message. That’s the at-least-once deal. Auto-commit is worse, because it commits on a timer that knows nothing about your handler.

confluent-kafka defaults to enable.auto.commit=true, auto.commit.interval.ms=5000 and enable.auto.offset.store=true. That last one stores an offset as soon as poll() or consume() hands you the message, not when you finish it. Take 100, and the background commit can mark all 100 done while you’re on message 20. Die then, and 80 are gone.

It’s ticking off your whole shopping list at the store entrance. Get called away in aisle 3, and the list still says you’re done. (The Java client commits inside poll(), which its javadoc calls at-least-once if you finish each batch first.)

Why does a Kafka consumer get duplicates after a rebalance?

Because a partition moves to its new owner at the last committed offset. Anything the old owner processed but didn’t commit runs again. The nasty case is a slow handler. Miss max.poll.interval.ms (300 s by default) and the consumer is thrown out of the group. The handler doesn’t know. It finishes, and it writes.

The test harness ran three consumers with max.poll.interval.ms=10000. One message in 500 stalled for 12 seconds the first time it was handled, and librdkafka logged:

%4|1790766143.381|MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (10000ms) exceeded by 42ms (adjust max.poll.interval.ms for long-running message processing): leaving group

Across five runs that made 50 evictions. Guess first: how many became duplicate credits?

All 50. Each zombie finished its credit within two seconds, and its offset commit failed with UNKNOWN_MEMBER_ID. By then another consumer had the same message. In production, that’s 50 customers with the wrong balance, and no error anywhere. With the inbox: 0 duplicates and 50 “skipped duplicate” log lines. KIP-848’s newer group protocol, GA since Kafka 4.0, doesn’t change this.

Try it: python measure_rebalance.py --variant b --runs 5, then --variant c.

The producer, or your outbox relay, sent it twice

Producer idempotence ends with the session. An outbox relay that publishes a row and dies before marking it sent publishes it again after a restart: new offset, same event. Only the event ID links the copies, so that’s what the consumer dedups on. Part 3 covers the outbox relay side.

Lock and visibility timeouts

Kafka’s share groups, production-ready since 4.2, lock each record for share.record.lock.duration.ms (30 s by default), then hand it to someone else. SQS redelivers when the visibility timeout (also 30 s by default) runs out. Either way, the first handler may still be running when the second copy arrives.

Which of these four have you seen? I’d bet on the rebalance. It shows up under load, exactly when nobody is reading the logs.

How Segment deduplicated billions of messages

Segment (now part of Twilio) published numbers for a different one: producers sending twice. In Delivering billions of messages exactly once (June 2017), Amir Abu Shareb traces its duplicates to clients that retry. Someone books a room in the HotelTonight app on a bus, and the bus goes into a tunnel mid-upload. The app retries events the server already had: the two generals, on a phone.

Segment’s server metrics put repeats at about 0.6% of the events it took in within a four-week window. Sounds small. But for a store with billions in revenue, Segment points out, 0.6% can decide between a profit and a loss of millions.

Its fix has this post’s shape. Each event carries a messageId, usually a UUIDv4, like the event_id header in publish.py below. A dedup worker keeps seen IDs in RocksDB, and commits the input offset only after RocksDB persists the key. Three months in, 200 billion messages had gone through it, and RocksDB held about 60 billion keys, 1.5 TB on disk.

The designs part ways on a crash. Segment’s worker repairs its output topic until it agrees with RocksDB. handle_once commits the key and the credit together, so there’s nothing to repair.

How do you implement the idempotent consumer pattern in Python?

Give every message a producer-set ID. In one Postgres transaction, insert that ID into a processed_messages table with ON CONFLICT DO NOTHING. Apply the side effect only if a row came back. Commit the Kafka offset after the transaction commits. That’s the pattern; the rest is detail.

Think of the stamp on your hand at a gig. You get it as you pay, so walking back in doesn’t cost you twice. Stamp people later, in a separate step, and somebody pays twice. Strictly, the stamp is deduplication. Idempotency is the result: running twice has the same effect as running once.

• Dedup on the producer’s event ID, in a header (or CloudEvents source plus id), scoped per consumer. Not the offset, which changes on a republish. It does the job an idempotency key does for an HTTP API.
• Database first, offset second. The reverse is at-most-once.
• No distributed lock. When a zombie and the new owner race, the primary key serializes them: the second INSERT waits, then gets no row.
• Errors. No event ID means nothing to dedup on, so the message goes to a dead-letter topic. An exception rolls back the inbox row with everything else.

The table, from the repo’s schema.sql (which also holds the demo’s accounts and ledger):

-- The inbox: one row per message this consumer has fully processed.
CREATE TABLE IF NOT EXISTS processed_messages (
    consumer     text        NOT NULL,              -- which consumer (group) processed it
    message_id   text        NOT NULL,              -- the producer's event ID, not the offset
    processed_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer, message_id)
);

And the complete consume.py:

"""An idempotent Kafka consumer: the inbox row and the side effect commit in one
Postgres transaction, and the Kafka offset is committed after that.

    python consume.py        # reads consumer.deposits, see README.md
"""
import json
import logging
import os

import psycopg
from confluent_kafka import Consumer, KafkaException, Message, Producer

DSN = os.environ.get("CONSUMER_DSN", "postgresql://patterns:patterns@localhost:55432/consumer")
BROKERS = os.environ.get("KAFKA_BROKERS", "localhost:59092")
CONSUMER_NAME = "wallet-credits"  # scopes the inbox: every consumer dedups on its own
log = logging.getLogger("consume")


def credit_account(conn: psycopg.Connection, event_id: str, event: dict) -> None:
    """The side effect. It's an increment, so running it twice credits twice."""
    conn.execute("UPDATE accounts SET balance = balance + %s WHERE id = %s",
                 (event["amount"], event["account_id"]))
    conn.execute("INSERT INTO ledger (event_id, account_id, amount) VALUES (%s, %s, %s)",
                 (event_id, event["account_id"], event["amount"]))


def handle_once(conn: psycopg.Connection, event_id: str, event: dict, apply=credit_account) -> bool:
    """Apply the event unless this consumer already has. Returns True if it ran."""
    with conn.transaction():
        claimed = conn.execute(
            "INSERT INTO processed_messages (consumer, message_id) VALUES (%s, %s) "
            "ON CONFLICT DO NOTHING RETURNING message_id",
            (CONSUMER_NAME, event_id),
        ).fetchone()
        if claimed is None:
            return False  # already done: skip it, but still commit the offset
        apply(conn, event_id, event)  # an exception here rolls back the inbox row too
    return True


def event_id_of(msg: Message) -> str | None:
    for key, value in msg.headers() or []:
        if key == "event_id" and value:
            return value.decode()
    return None


def run(topic: str, group: str, apply=credit_account, **overrides) -> None:
    consumer = Consumer({
        "bootstrap.servers": BROKERS,
        "group.id": group,
        "enable.auto.commit": False,      # we commit, and only after Postgres has
        "auto.offset.reset": "earliest",  # a new group starts at the beginning, not the end
        **overrides,
    })
    dead_letters = Producer({"bootstrap.servers": BROKERS})
    consumer.subscribe([topic])
    with psycopg.connect(DSN, autocommit=True) as conn:
        try:
            while True:
                msg = consumer.poll(1.0)
                if msg is None:
                    continue
                if msg.error():
                    log.warning("consumer error: %s", msg.error())
                    continue
                event_id = event_id_of(msg)
                if event_id is None:  # nothing to dedup on, so park it instead of guessing
                    dead_letters.produce(topic + ".dlq", value=msg.value(), key=msg.key(),
                                         headers=msg.headers())
                    dead_letters.flush()
                elif not handle_once(conn, event_id, json.loads(msg.value()), apply):
                    log.info("skipped duplicate %s", event_id)
                try:
                    consumer.commit(message=msg, asynchronous=False)  # last, after the DB commit
                except KafkaException as e:
                    # We lost this partition in a rebalance. Its new owner will get the
                    # message again and skip it, because the inbox row is already there.
                    log.warning("offset commit failed: %s", e)
        finally:
            consumer.close()


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
    run("consumer.deposits", group=CONSUMER_NAME)

Three details matter. auto.offset.reset defaults to latest, so a new group would skip what’s already in the topic. The skip path still commits the offset, or the duplicate comes back forever. And a failed commit is logged, not raised: a rebalance took the partition, and the new owner will find the inbox row.

The producer side, from publish.py:

# publish.py, without its docstring and demo __main__
import json
import os

from confluent_kafka import Producer

BROKERS = os.environ.get("KAFKA_BROKERS", "localhost:59092")


def make_producer() -> Producer:
    # librdkafka (and so confluent-kafka) defaults enable.idempotence to false
    return Producer({"bootstrap.servers": BROKERS, "enable.idempotence": True})


def publish(producer: Producer, topic: str, event_id: str, event: dict) -> None:
    producer.produce(topic, key=str(event["account_id"]), value=json.dumps(event),
                     headers={"event_id": event_id})

Now the price. Guess first: what does that extra insert cost per message?

About 7% in my runs. On 100,000 messages with one consumer, the loop did 323 messages/s with the inbox and 347.8 without (medians of 5 runs). The no-inbox runs spread from 314.6 to 365.1, so call it rough. Auto-commit ran at 445.1 messages/s; the next section shows what that speed costs.

Try it: python measure_throughput.py --runs 5. Every “Try it” runs in this post’s repo folder after the README’s setup. The numbers are from my runs, so yours will differ.

Then the evt-2 crash. The pytest suite crashes a simulated queue, then the real consumer, with os._exit() right after the database commit. Double delivery, the crash and two racing consumers all end with one credit.

python -m pytest -q
# Output
# .............                                                            [100%]
# 13 passed in 19.32s

Without the inbox, evt-2 lands in the ledger twice: account 1 ends at 600, not 500. With it, the restarted consumer logs skipped duplicate evt-2. Offset-first loses the message outright.

The inbox pattern: an idempotent consumer with a transaction

The inbox pattern is the idempotent consumer pattern done with a transaction. Microsoft’s pattern page puts it directly: when the dedup marker and the side effects commit together, that variant is the Inbox pattern. It’s the consumer-side companion to the transactional outbox, and what we just built.

The idempotent consumer pattern goes by other names: idempotent consumer on Chris Richardson’s microservices.io, and idempotent receiver in Enterprise Integration Patterns (Apache Camel’s Idempotent Consumer EIP). In .NET it’s the transactional inbox, built into NServiceBus and MassTransit.

A store-then-process variant saves the raw message, commits the offset, and lets a worker process rows later. It pays off when handlers are slow enough to hit max.poll.interval.ms. The price is a second poller, and ordering you manage yourself.

What happens when you kill the consumer mid-stream?

Without the inbox, every restart replays work, and auto-commit with batches loses messages too. With the inbox in the side effect’s transaction, 100 hard kills gave no duplicate credits and no lost messages.

Each run publishes 10,000 credit events to a 6-partition topic. The harness runs the consumer as a subprocess and kills it with Popen.kill(), a hard kill: no finally, no close(), no last commit. That’s evt-2’s crash, 20 times a run, each after 1 to 8 random seconds of work. A final consumer drains the topic, and the harness counts ledger rows per event ID.

Five runs per variant share one seeded kill schedule. Handlers sleep 10 ms to stand in for real work. session.timeout.ms=6000 spots a dead consumer in about 6 s instead of 45, with librdkafka’s default classic protocol.

Guess first. Variant a1 is auto-commit with the defaults, one message per poll(). Twenty kills, 10,000 credits: how many get applied twice? Fifty? Five hundred?

A median of 3,648 per run. Here’s what each variant did:

• a1 replayed up to 5 s of work per kill: 18,106 duplicates across 100 kills. It lost nothing, because only the message in hand had its offset stored early.
• a2 stores a whole batch’s offsets when consume() returns it. That’s the shopping list: it lost messages in all five runs, 391 in total.
• b duplicated only when a kill landed between the two commits: 20 across 100 kills. Small isn’t zero. Each one is a customer credited twice.
• c had no duplicates and no losses, and every balance came out exact.

Try it: python measure_kill.py --variant c --runs 5, then a1, a2 and b.

Variant b looks almost good enough, until you remember the zombie. It doubled all 50 credits without a crash, and committing after the database didn’t help. Only the inbox caught them.

What my lab can’t show: network partitions. This was one broker on one machine. Your a and b counts will differ; the c result won’t.

Do: turn off auto-commit, and commit each offset after the transaction that recorded its ID. Even an idempotent handler can’t bring back a2’s lost messages.

What if the side effect isn’t in your database?

Then no single transaction covers it, so push idempotency downstream. Pass the provider a key derived from the message ID, so its dedup catches your retry. If it has no keys, record each attempt and check the provider’s status before retrying.

• Pass an Idempotency-Key such as f"{CONSUMER_NAME}:{event_id}", identical on every redelivery. Part 1 covers the server side of the Idempotency-Key header.
• Processing and done states, with a lease. Insert the inbox row as processing with a locked_until, call out, mark it done. On redelivery, done means skip. An expired lease means retry with the same key, or ask the provider whether the first call landed.
• Inbox, then outbox. Write an outbox row inside the inbox transaction, and let the relay make the call with the same key.

Think of a registered letter. You write “posting it” in your log before you leave, and “posted” when you get back. If the log only says “posting it”, you check the tracking number before you post another.

And with neither keys nor a status lookup? No clever fix exists. You choose between a rare duplicate (retry) and a rare loss (don’t), whichever costs less.

Email often has no key at all. Postmark’s support center says it doesn’t currently support an idempotency key, and points you to HTTP response codes, API error codes and webhooks instead.

Here’s how I’d fit that to the two states. Put the event ID in the email’s Metadata. Postmark’s delivery webhook hands it back with the MessageID, so it can mark the row done. If a call times out, let the lease expire and look again. You’ll still get the rare duplicate or loss, just less often.

How a read-only Redis kept Twilio recharging customers’ cards

Back to Twilio’s cards from the top. Its 2013 billing postmortem shows what happens when a call lands and its record doesn’t. A network loss made every Redis replica request a full resync at once. Engineers restarted the overloaded master. It read the wrong config file, dropped all balance data and came back read-only.

Usage kept triggering auto-recharges against zero balances that couldn’t be updated, so recharges were attempted multiple times. Twilio refunded the charges and credited each affected account 10% of its last 30 days of spend.

The charge was the side effect outside the database, checked against a Redis balance. One of Twilio’s fixes is the rule processing gives you: if balances don’t exist or can’t be written, don’t charge cards. No committed row, no call. And a row stuck in processing means asking the provider, not trusting a balance that reads zero.

How long should you remember message IDs?

For the longest redelivery that can happen, plus a margin. That’s your worst replay: a consumer reset to the start of a topic, a dead-letter queue replayed weeks later, a backfill. Delete an ID too early, and the duplicate gets through.

A default Kafka broker keeps topics 7 days (log.retention.hours=168) and drops an empty group’s offsets after 7 days (offsets.retention.minutes=10080). Then auto.offset.reset=earliest, as in consume.py, replays the whole week. So the inbox had better hold a week of IDs.

How big is that? Say you take in 5 million messages a day (a made-up figure). Fourteen days, a week plus a week of margin, is 70 million rows. Guess first: how many bytes does one remembered ID cost?

In my runs, 194.5 bytes as text and 118.1 with uuidv7(). Text costs 45% more than a uuid column, and uuidv7() shrinks the index another 24%. Random keys get filed into random drawers and keep splitting pages. Time-ordered keys always go at the back.

Try it: python measure_storage.py --runs 5. The 70 million row column is my arithmetic, and it leaves out the cleanup index below.

Cleanup is a cron job that repeats a batched delete until nothing’s left. From retention.sql:

-- Once: lets the cleanup find old rows without scanning the whole table.
CREATE INDEX IF NOT EXISTS processed_messages_processed_at ON processed_messages (processed_at);

-- From cron, repeated until it deletes 0 rows. Small batches keep locks and WAL bursts short.
DELETE FROM processed_messages
WHERE ctid IN (
    SELECT ctid FROM processed_messages
    WHERE processed_at < now() - interval '14 days'
    LIMIT 10000
);

Try it: python -m pytest tests/test_retention.py -q. From 25,000 expired rows and 100 recent ones, the loop deletes 10,000, 10,000, 5,000, then 0.

Dropping partitions would be cheaper, but Postgres wants the partition key in the primary key (“unique constraint on partitioned table must include all partitioning columns”). So partition by the message ID: with UUIDv7, a range of IDs is a range of time. The outbox table has the same cleanup problem.

How long can a sender hand you the same message?

Longer than a week, for some. Here’s each sender’s window, and what it gives you to dedup on:

So take the longest window among your sources, and add margin. Is 14 days enough? Mostly, yes. It covers Stripe’s automatic retries, Shopify, GitHub’s redelivery and a default Kafka topic. It misses Stripe’s Dashboard resend by a day.

The long tail needs more. A CLI resend on day 30, a Pub/Sub seek a month back, or an SQS FIFO message redriven from its DLQ four weeks later all need 31 days plus margin. That’s 38 days: 190 million rows, about 22.4 GB with uuidv7() keys (my arithmetic).

Segment kept a four-week window, capped by disk size rather than time, and paged on-call if a traffic spike squeezed it under 24 hours. I’d copy that alarm even with a time-based delete: page when now() - min(processed_at) drops below your longest window.

When you can skip the table

When running the handler twice already gives the same result:

• Upserts by business key. “Set order 42 to shipped” can run ten times. “Add $1 to account 7” can’t.
• Version checks. Apply an event only if it’s newer than the row, like the fencing token in the next post on Redis locks.
• Re-indexing documents, upserted by document and chunk ID, the way a RAG ingestion pipeline should do it.

Broker dedup windows are no substitute: 5 minutes in SQS FIFO, 10 in Service Bus, 100 seconds (or 100 IDs per producer) in Redis Streams.

Do AI agents and LLM gateways need the idempotent consumer pattern?

Yes, wherever they react to a message, because the queue underneath delivers at least once too:

• Spend events. The LLM gateway architecture post keeps spend in a ledger. A consumer that adds each request’s cost to a budget is “add $1 to account 7” again. Give it an inbox.
• Tool calls from a queue. An agent step that runs send_invoice per message sends a second invoice on redelivery. Use processing and done states, keyed on the message ID.
• Retries after a 429. A worker backing off on a rate limit can outlast its visibility timeout, so the job goes to someone else. Part 2’s rate limiting algorithms explain the 429; the inbox catches the second copy.

Idempotent consumer pattern mistakes to check before you ship

Each passes the happy-path test: one consumer, no crashes, no rebalances. They show up on the first bad deploy.

• Auto-commit on. A median of 3,648 doubled credits per 20 kills, and batches lost 63 messages.
• Offset first. At-most-once: test_offset_first_is_at_most_once loses evt-1 for good.
• Dedup on the offset. A relay resend gets a new offset. Use the event ID.
• “We commit after the database.” Still 20 doubled credits in 100 kills, and all 50 zombie ones.
• Retention from a round number. 14 days misses Stripe’s 15-day resend.

If you do one thing this week, search your consumers for enable.auto.commit. Unless it says false, it’s on: that’s confluent-kafka’s default.

And evt-2? It still arrives twice, and nothing in Kafka can stop that. But with the idempotent consumer pattern, the second copy finds its row in processed_messages, logs a skip and commits its offset. Account 1 gets its 100 once. If I had to guess where teams slip first, it’s retention: nothing fails until someone replays a week-old topic into an inbox that remembers three days.

Over to you: what’s the longest redelivery window among your sources, and did you set inbox retention from it or from a round number? Tell me in the comments.

The consumer, the kill harness and the 13 tests are on GitHub. Found a crash they miss? Open an issue or a pull request.

Next in this series: Redis distributed locks, and why a lock alone can’t stop a zombie writer. Browse the system design hub meanwhile, and join the newsletter to get the next one.