Streaming data: Kafka, exactly-once, and why it’s harder than it sounds

Imagine an e-commerce platform processing thousands of orders every second. A customer places an order, an event enters a Kafka topic, and a consumer processes that event to create a purchase record.

Everything works perfectly until the consumer crashes.

The purchase record was written successfully, but the consumer never committed its Kafka offset. When the application restarts, it reads the same event again and creates another purchase record.

From Kafka’s perspective, nothing went wrong. It did exactly what a reliable messaging system should do: it made sure an unacknowledged unit of work could be retried.

From the application’s perspective, however, one order has become two purchases.

This is the central challenge of reliable stream processing: receiving a message once, processing it once, and producing exactly one observable business effect are three different problems.

Apache Kafka provides powerful mechanisms for controlling duplicates, coordinating transactions, and recovering from failures. But understanding those mechanisms requires knowing precisely where their guarantees begin and end.

In this tutorial, we’ll explore:

• How Kafka stores and delivers events, and why offsets matter.

• The differences between at-most-once, at-least-once, and exactly-once processing.

• Why producer idempotence doesn’t solve consumer-side duplicates.

• How Kafka transactions coordinate output records and consumed offsets.

• A runnable Python producer and consumer that deliberately reproduces duplicate processing.

• An idempotent database implementation that survives the same failure.

• What Kafka’s guarantees mean when external databases, APIs, and real-world side effects enter the picture.

We’ll use Apache Kafka 4.3.1 and the Confluent Python client 2.16.0, both current releases as of October 2026. Kafka 4.3.1 is part of the supported Apache release line, while the Python client version was released on October 7, 2026.

1. Understanding Kafka’s delivery model

Kafka is a distributed event log.

Unlike a traditional work queue where a message may disappear after acknowledgment, Kafka stores records in an ordered, append-only log divided into partitions.

Consider a topic containing customer orders:

Topic: orders

Partition 0

Offset 0

A100

Offset 1

A101

Offset 2

A102

Offset 3

A103

Committed offset: 2

Next record: offset 2

The committed offset identifies the next position to consume, not the last record successfully processed.Each record contains information such as:

• Key: An optional value used by the default partitioner to choose a partition.

• Value: The event payload, typically JSON, Avro, Protobuf, or another serialization format.

• Partition: The ordered log containing the record.

• Offset: The record’s position within that partition.

• Timestamp: Metadata associated with the event.

Offsets are unique within a topic partition, not across the entire topic.

For example, partition 0 and partition 1 can both contain a record at offset 42.

Consumers and consumer groups

Applications read records using consumers.

Multiple consumers can cooperate through a consumer group, sharing the partitions assigned to that group. Within an ordinary consumer group, a partition is assigned to one member at a time under normal operation.

A consumer tracks two important positions:

1. Its current position, indicating the next record it will fetch or return.

2. Its committed position, which the group can use to resume processing after a restart or rebalance.

These positions can differ.

A consumer may have fetched offset 42 and successfully executed its business logic while its last committed position remains 42. If the process crashes, that record can be delivered again.

This is not necessarily a bug. It is a fundamental consequence of separating message consumption from acknowledgment of completed work.

Why replay is a feature

Kafka does not normally delete a record simply because a consumer has read it.

Records remain available according to the topic’s retention or compaction configuration. Different consumer groups can independently consume the same data, and applications can deliberately replay retained events.

This makes Kafka useful for event-driven microservices, analytics pipelines, audit streams, and rebuilding application state.

But replayability has an important implication:

Your processing logic must be designed for the possibility that an event will be encountered more than once.

2. The three delivery guarantees

These guarantees are often described as a simple hierarchy. In practice, their meaning depends on which operations and systems are included.

The crucial phrase is within a defined transactional scope.

At-most-once: commit before processing

Suppose the consumer commits the offset first and then writes to a database.

The sequence looks like this:

Read event

Commit offset

Write result

A crash after the offset commit but before the database write can permanently skip the business operation.

The consumer has already told Kafka that it can resume from the next event.

When the process restarts, the original event is not automatically retried by that consumer group.

This strategy is acceptable for some disposable telemetry, sampling, or approximate analytics workloads. It is usually inappropriate when every business operation matters.

At-least-once: process before committing

Now reverse the order:

Read event

Write result

Commit offset

A crash after the database write but before the offset commit causes the event to be replayed.

This is generally safer when losing work is unacceptable.

However, the database modification and Kafka offset commit remain separate operations. A crash between them creates a duplicate-processing window.

Exactly-once: coordinate the result and the progress

To produce exactly one committed result, the system must establish a stronger invariant:

The processing result is committed if and only if the corresponding progress is committed.

Kafka can provide that coordination for certain read-process-write workflows when both the output and consumer offsets are committed in the same Kafka transaction.

But an arbitrary external database or HTTP service does not automatically participate in that transaction.

This distinction will become especially important in our Python experiment.

3. Producer idempotence solves only part of the problem

Kafka supports idempotent producers, which protect against duplicate log entries caused by certain producer retries.

Consider this sequence:

1. A producer sends an event to a broker.

2. The broker successfully appends the event.

3. The acknowledgment is lost during a network interruption.

4. The producer cannot tell whether the original write succeeded.

5. The producer retries.

Without idempotence, the broker may append a second copy of the same event.

With idempotence enabled, Kafka tracks producer identity and sequence information so that retries of the same producer batch do not create duplicate log records.

The relevant producer setting is enable.idempotence=true.

Kafka’s current producer configuration enables idempotence by default when no conflicting settings are supplied. Idempotence requires acknowledgments from all in-sync replicas, retries enabled, and no more than five in-flight requests per connection.

What idempotence does not guarantee

Producer idempotence does not deduplicate arbitrary business events.

For example, suppose an application deliberately calls its producer twice with the same JSON payload. Those calls are two distinct sends, and Kafka can legitimately store both records.

Likewise, producer idempotence does not stop a consumer from re-executing a database update after a crash.

This gives us three distinct duplicate scenarios:

Another subtle point: an event ID is not the same thing as a Kafka offset.

An event ID identifies a logical business event, while an offset identifies one record’s location in Kafka. The same logical event can appear at multiple offsets if it is independently published more than once.

That distinction determines how we’ll implement our fix.

4. Set up a reproducible Kafka environment

We’ll build a small experiment using:

• Apache Kafka 4.3.1 running locally in Docker.

• Python 3.11 or newer.

• The confluent-kafka Python package.

• SQLite as a durable local database.

• Two topics that let us compare unsafe and safe processing independently.

The example uses synthetic order events. It does not connect to a real payment processor or charge anyone.

Step 1: Start Kafka

Ensure Docker is installed and running, then execute:

docker run -d \
  --name kafka-lab \
  -p 9092:9092 \
  apache/kafka:4.3.1

The official Apache image supports a single-node development setup with its default configuration. Kafka 4.x uses KRaft rather than ZooKeeper, so we do not need to start a separate ZooKeeper container.

The container exposes Kafka at localhost:9092.

This is a development configuration, not a fault-tolerant production deployment. There is only one broker, one copy of our data, and no broker-level redundancy.

To inspect its startup logs:

docker logs kafka-lab

If the container is still initializing, allow startup to finish before running the producer.

Step 2: Create a Python environment

Create an empty project directory and a virtual environment:

mkdir kafka-exactly-once-lab
cd kafka-exactly-once-lab

python -m venv .venv
source .venv/bin/activate

python -m pip install confluent-kafka==2.16.0

The activation command is for bash-compatible shells on Linux and macOS. On Windows, use the corresponding activation script for your shell.

The Python client uses librdkafka underneath and exposes producers, consumers, and administrative operations. We won’t need to write our own network protocol implementation or Kafka serialization layer.

Step 3: Understand the experiment

We’ll use two independent topics and consumer groups:

Each experiment follows the same sequence:

1. Produce one order event.

2. Consume the event.

3. Commit a SQLite database modification.

4. Deliberately terminate the process before committing its Kafka offset.

5. Restart the consumer.

6. Inspect the resulting database state.

The only significant difference is that the safe consumer records which business events it has already applied, inside the same SQLite transaction as the associated database modification.

The application creates its Kafka topics automatically using Kafka’s administrative API, so no separate topic-creation commands are required.

5. Build the runnable producer and consumer

Create a file named demo.py with the complete implementation below.

import argparse
import json
import os
import sqlite3
import time

from confluent_kafka import Consumer, KafkaError, KafkaException, Producer
from confluent_kafka.admin import AdminClient, NewTopic

BROKER = "localhost:9092"
DB_FILE = "orders.sqlite3"


def topic_for(mode):
    return f"orders-{mode}"


def create_topic(topic):
    admin = AdminClient({"bootstrap.servers": BROKER})
    futures = admin.create_topics(
        [NewTopic(topic, num_partitions=1, replication_factor=1)]
    )

    try:
        futures[topic].result(timeout=15)
    except KafkaException as exc:
        if exc.args[0].code() != KafkaError.TOPIC_ALREADY_EXISTS:
            raise


def publish(mode):
    topic = topic_for(mode)
    create_topic(topic)

    event = {
        "event_id": f"order-{mode}-001",
        "amount_cents": 2599,
    }

    producer = Producer({
        "bootstrap.servers": BROKER,
        "enable.idempotence": True,
        "acks": "all",
    })

    failures = []

    def delivered(error, message):
        if error:
            failures.append(str(error))
        else:
            print(
                f"Produced {event['event_id']} "
                f"at offset {message.offset()}"
            )

    producer.produce(
        topic,
        key=event["event_id"].encode(),
        value=json.dumps(event).encode(),
        on_delivery=delivered,
    )

    remaining = producer.flush(15)

    if remaining or failures:
        raise RuntimeError(
            f"Delivery failed: {failures}, pending={remaining}"
        )


def init_database():
    with sqlite3.connect(DB_FILE) as db:
        db.execute("""
            CREATE TABLE IF NOT EXISTS charges (
                id INTEGER PRIMARY KEY,
                event_id TEXT NOT NULL,
                amount_cents INTEGER NOT NULL
            )
        """)

        db.execute("""
            CREATE TABLE IF NOT EXISTS processed_events (
                event_id TEXT PRIMARY KEY
            )
        """)


def apply_event(mode, event):
    with sqlite3.connect(DB_FILE) as db:
        db.execute("BEGIN IMMEDIATE")

        if mode == "safe":
            result = db.execute(
                """
                INSERT OR IGNORE INTO processed_events (event_id)
                VALUES (?)
                """,
                (event["event_id"],),
            )

            if result.rowcount == 0:
                print(
                    "Already processed; skipping duplicate",
                    flush=True,
                )
                return

        db.execute(
            """
            INSERT INTO charges (event_id, amount_cents)
            VALUES (?, ?)
            """,
            (event["event_id"], event["amount_cents"]),
        )

        print("Created charge row", flush=True)


def consume(mode, crash):
    init_database()

    consumer = Consumer({
        "bootstrap.servers": BROKER,
        "group.id": f"demo-{mode}",
        "auto.offset.reset": "earliest",
        "enable.auto.commit": False,
        "enable.auto.offset.store": False,
        # A crashed consumer keeps its group membership until its session expires
        # (45 s by default). Keep the session short so the restarted consumer is
        # assigned the partition within seconds instead of waiting out the crash.
        "session.timeout.ms": 6000,
        "heartbeat.interval.ms": 2000,
    })

    consumer.subscribe([topic_for(mode)])

    try:
        deadline = time.monotonic() + 60

        while time.monotonic() < deadline:
            message = consumer.poll(1.0)

            if message is None:
                continue

            if message.error():
                raise KafkaException(message.error())

            event = json.loads(message.value())

            print(
                f"Received {event['event_id']} "
                f"at offset {message.offset()}",
                flush=True,
            )

            apply_event(mode, event)

            print("SQLite commit complete", flush=True)

            if crash:
                print(
                    "Simulated crash before Kafka offset commit",
                    flush=True,
                )
                os._exit(23)

            consumer.commit(
                message=message,
                asynchronous=False,
            )

            print("Kafka offset committed", flush=True)
            return

        raise TimeoutError("No message received within 60 seconds")

    finally:
        consumer.close()


def inspect():
    init_database()

    with sqlite3.connect(DB_FILE) as db:
        rows = db.execute("""
            SELECT event_id, COUNT(*), SUM(amount_cents)
            FROM charges
            GROUP BY event_id
            ORDER BY event_id
        """).fetchall()

    for event_id, count, total in rows:
        print(
            f"{event_id}: rows={count}, total_cents={total}"
        )


def main():
    parser = argparse.ArgumentParser()

    commands = parser.add_subparsers(
        dest="command",
        required=True,
    )

    produce_parser = commands.add_parser("produce")
    produce_parser.add_argument(
        "mode",
        choices=["unsafe", "safe"],
    )

    consume_parser = commands.add_parser("consume")
    consume_parser.add_argument(
        "mode",
        choices=["unsafe", "safe"],
    )
    consume_parser.add_argument(
        "--crash",
        action="store_true",
    )

    commands.add_parser("inspect")

    args = parser.parse_args()

    if args.command == "produce":
        publish(args.mode)
    elif args.command == "consume":
        consume(args.mode, args.crash)
    else:
        inspect()


if __name__ == "__main__":
    main()

The implementation intentionally handles one message per consumer invocation, making the replay behavior easy to observe.

For a production consumer, we’d normally process continuously, coordinate partition assignments, handle malformed events, apply retry and backoff policies, and manage offsets across multiple partitions.

Here, the one-record design helps isolate the specific correctness problem.

Key design decisions

Several details are essential to the experiment.

We explicitly disable automatic offset commits.

The enable.auto.commit setting is false. This ensures our consumer won’t automatically acknowledge progress while we’re trying to reproduce the crash.

We disable automatic offset storage.

The enable.auto.offset.store setting is also false. This prevents the client from implicitly advancing its locally stored commit position as records are returned to the application.

We commit offsets synchronously after processing.

The consumer calls commit(message=message, asynchronous=False) only after SQLite processing succeeds. Passing the message commits the offset immediately after that record, rather than blindly committing an unrelated position.

The current Confluent Python client documents this explicit manual-commit approach.

We force an abrupt process termination.

Calling os._exit(23) ends the Python process immediately, bypassing normal exception handling and cleanup.

Exit code 23 is an arbitrary marker chosen for this demonstration. It is not a Kafka error code.

We shorten the consumer session.

A crashed consumer does not leave its group cleanly: the broker keeps its membership until session.timeout.ms expires, 45 seconds by default. Until then the partition stays assigned to the dead process, so a consumer restarted immediately sits in a rebalance and receives nothing. We verified this against Kafka 4.3.1: with the defaults, the restarted consumer timed out without receiving the order. Setting session.timeout.ms to 6000 (the broker’s default minimum) and heartbeat.interval.ms to 2000 lets the restart take over within seconds. In production, the same effect explains why redelivery after a crash is delayed by up to one session timeout.

That gives us a reproducible failure point: the database change has committed, but the consumer has not sent its offset commit.

We keep the database durable across consumer restarts.

SQLite writes to a file called orders.sqlite3. Restarting the Python process does not erase that file.

This matters because the experiment would teach us very little if the simulated crash also removed the effect we were trying to deduplicate.

6. Reproduce the duplicate-delivery problem

Now we’ll deliberately break the unsafe consumer.

For the cleanest results, run each sequence once against the new topics and database created by this tutorial.

Step 1: Publish an order

Run:

python demo.py produce unsafe

On a fresh topic, the output should resemble:

Produced order-unsafe-001 at offset 0

We’ve produced one event representing an order worth 2,599 cents.

The message has a stable event ID, but the unsafe consumer will not use that ID to prevent duplicate database writes.

Step 2: Process the order and simulate a crash

Run:

python demo.py consume unsafe --crash || test "$?" -eq 23

The command accepts the intentional exit code 23 while still exposing unexpected failures.

Expected output:

Received order-unsafe-001 at offset 0
Created charge row
SQLite commit complete
Simulated crash before Kafka offset commit

Notice the ordering.

SQLite has already committed the new row, but Kafka has not received an offset commit.

Our consumer group therefore has no recorded progress for this topic partition.

If we inspected SQLite immediately, there would be one charge record.

Step 3: Restart the consumer

Run:

python demo.py consume unsafe

Expected output:

Received order-unsafe-001 at offset 0
Created charge row
SQLite commit complete
Kafka offset committed

Kafka delivered the same record at offset 0.

The consumer created another database row, because nothing told it that the business operation had already been performed.

Step 4: Inspect the result

Run:

python demo.py inspect

Expected result:

order-unsafe-001: rows=2, total_cents=5198

We produced one Kafka event but created two database records.

Unsafe consumer: two processing attempts

First attempt

Database row inserted

Offset not committed

Second attempt

Another row inserted

Offset committed

Final database result

2 rows · 5,198 cents

The important conclusion is that the producer’s idempotence setting did not prevent the duplicate.

Our producer behaved correctly. There was one record in Kafka. The duplication happened entirely within the consumer’s interaction with the external database.

This is why simply enabling idempotent production is insufficient for end-to-end correctness.

7. Fix the consumer with an idempotency key

Now we’ll repeat the experiment using the safe processing mode.

The difference is a simple but powerful database invariant:

A logical event may create its business effect only if its event ID has not already been processed.

Our database contains two tables.

The charges table records the business effect. The processed_events table acts as a durable inbox, recording which event IDs have already been applied.

The safe processing function performs the following operations inside one SQLite transaction:

1. Attempt to insert the event ID into processed_events.

2. If the insert succeeds, create the corresponding charge row.

3. If the ID already exists, skip the charge.

4. Commit the transaction.

The database’s primary-key constraint enforces uniqueness. The transaction makes the deduplication record and business write atomic.

The critical detail is that we do not perform a separate existence check and then blindly write the result. Such a pattern would introduce a race between concurrent consumers.

Step 1: Produce the safe event

python demo.py produce safe

Expected output for the fresh topic:

Produced order-safe-001 at offset 0

Step 2: Simulate the same crash

python demo.py consume safe --crash || test "$?" -eq 23

Expected output:

Received order-safe-001 at offset 0
Created charge row
SQLite commit complete
Simulated crash before Kafka offset commit

This failure is identical to the unsafe version.

The consumer has written its database state but has not committed its Kafka offset.

Step 3: Restart the consumer

python demo.py consume safe

Expected output:

Received order-safe-001 at offset 0
Already processed; skipping duplicate
SQLite commit complete
Kafka offset committed

Kafka again delivers the same event.

This time, however, the consumer finds that the event ID already exists in the processed_events table.

The business effect is skipped, and the Kafka offset is committed successfully.

Step 4: Inspect both experiments

python demo.py inspect

Expected output:

order-safe-001: rows=1, total_cents=2599
order-unsafe-001: rows=2, total_cents=5198

Same crash, different database outcomes

Unsafe

Duplicate business effectIdempotent

One committed business effectBoth consumers received one event twice across separate executions.

The distinction is not in Kafka’s delivery behavior. It is in how our application handles replay.

Why the safe implementation works

Think through the possible crash points.

Case A: The process crashes before the database transaction commits.

The database rolls back any uncommitted changes. Kafka later redelivers the event, and the consumer can safely attempt the operation again.

Case B: The process crashes after the database transaction commits but before the Kafka offset commits.

Kafka redelivers the event. The deduplication key already exists, so the business modification is not repeated.

Case C: The process crashes after both commits succeed.

The business modification remains durable, and the consumer group normally resumes from the next committed position.

If Kafka later redelivers the same business event for another reason, the database constraint still prevents an additional effect.

The correctness invariant

For our synthetic example, the desired property is:

• For each valid, unique order event ID, the database contains at most one charge record created by this consumer.

• If processing successfully completes and the database remains available, the event eventually produces that one record.

That is stronger than merely hoping the consumer won’t receive duplicate messages.

It is an application-level guarantee enforced by a persistent database constraint.

There are still limits.

Our implementation assumes that event IDs are stable and uniquely identify the intended business operation. It also assumes that the database transaction succeeds or rolls back atomically.

If an event ID is accidentally reused for a different legitimate order, the second order could be skipped. Production systems therefore need clear rules for constructing IDs and deciding their scope.

For example, an order ID combined with an operation type may be appropriate when the same order legitimately triggers multiple distinct actions.

Why a primary key matters under concurrency

Imagine two workers encountering the same event simultaneously.

If both first query whether the event exists, both might observe that it is absent. Each could then attempt the business operation.

A unique constraint changes this behavior.

The database decides atomically whether each insert can succeed. Under our SQLite transaction model, only one attempt can establish ownership of that event ID.

The other receives the duplicate outcome and skips the business modification.

For a production database, it is generally preferable to handle the specific unique-key conflict rather than suppress unrelated database errors. SQLite’s broad INSERT OR IGNORE behavior is convenient for this minimal demonstration, but an explicit uniqueness-conflict policy is easier to audit in a larger application.

The overarching principle remains the same: deduplication and the business operation belong in the same atomic transaction.

8. What Kafka’s native exactly-once transactions actually guarantee

Our SQLite solution is useful, but it isn’t the same mechanism as Kafka’s native exactly-once processing.

Kafka transactions solve a more specific problem: coordinating the consumption of records from Kafka with the production of new records back into Kafka.

Consider a streaming pipeline:

Input topic → Transformation → Output topic

An application reads an order event, calculates a discount, and publishes a transformed event.

Without transactions, the following failure can occur:

1. The application consumes input record A.

2. It calculates output record B.

3. It successfully produces B.

4. It crashes before committing the consumed offset of A.

5. Another consumer restarts processing from A.

6. It produces B again.

An idempotent producer alone cannot solve this particular failure. The second attempt may run in another producer session and issue a new, legitimate send.

The transactional solution

Kafka provides a transactional producer API that lets the application coordinate its output records and consumer offsets.

The general workflow is:

1. Initialize a producer with a stable, appropriately scoped transactional ID.

2. Begin a Kafka transaction.

3. Read one or more input records.

4. Process the records and produce output messages within the transaction.

5. Add the corresponding input offsets to the transaction.

6. Commit the transaction.

The important property is that the produced output and consumed offsets become committed together.

If the transaction aborts, its produced records are not visible to consumers configured for committed reads, and the associated offset progress is not committed.

If the transaction commits, both the output records and the consumed offsets become committed.

The Kafka 4.3 documentation describes this coordinated read-process-write guarantee.

The relevant APIs

In the current Confluent Python client, transactional producers expose methods including:

• init_transactions() to initialize transaction state and handle previous producer instances using the same transactional ID.

• begin_transaction() to start a transaction.

• produce() to write records into that transaction.

• send_offsets_to_transaction() to include consumer progress in the transaction.

• commit_transaction() to commit the transaction.

• abort_transaction() to abandon a failed transaction.

For send_offsets_to_transaction(), the application supplies the next offsets to consume and the consumer group’s metadata.

The consumer must disable automatic commits for this pattern.

A robust transactional loop must also distinguish retriable failures, failures requiring transaction abort, and fatal failures. Blindly retrying every operation is not safe.

Why transactional IDs matter

A transactional ID provides continuity across instances of a logical producer.

Suppose the original process becomes temporarily disconnected. An orchestrator starts a replacement, but the original process later recovers and attempts to continue writing.

Without protection, both processes might believe they own the same processing task.

Kafka uses producer epochs and fencing to prevent an obsolete producer instance from continuing to commit transactions under an ownership identity that has been taken over.

This protects against so-called zombie producers.

Transactional IDs must therefore be assigned according to the ownership model of processing tasks, rather than reused arbitrarily across unrelated concurrent workers.

Read-committed consumers

Another essential piece is the consumer’s isolation level.

By default, Kafka consumers use read_uncommitted, which can expose records belonging to aborted transactions.

For transactional results, consumers should generally use isolation.level=read_committed.

With this isolation level:

• Nontransactional records remain readable.

• Records from committed transactions become visible.

• Records from aborted transactions are filtered out.

• Open transactions can delay the visibility of later records in the affected partition.

That last point is important for latency-sensitive systems. A long-running transaction can delay downstream consumers even when other records have already been appended to the log.

Kafka Streams simplifies the coordination

For Java applications, Kafka Streams provides a higher-level approach.

The current supported processing guarantee is processing.guarantee=exactly_once_v2.

Kafka Streams coordinates transactional production, committed offsets, and its managed processing state so applications don’t have to implement every transactional detail manually.

The default remains at_least_once; exactly-once processing must be enabled explicitly.

The older exactly_once and exactly_once_beta configuration values are deprecated. For a modern Kafka Streams application, use exactly_once_v2.

One important deployment detail: production exactly-once setups should use appropriately replicated internal topics and brokers. Kafka’s documentation recommends a three-broker deployment for durable transactional guarantees. A single-broker laptop environment is useful for demonstrating processing behavior, not for establishing production fault tolerance.

Exactly-once results do not mean exactly one execution

Here is a distinction that makes the entire subject easier to reason about.

A Kafka Streams application may execute processing logic again after a failed transaction.

The earlier attempt might have performed calculations, updated temporary state, or produced records into a transaction that later aborted.

After recovery, the same input may be processed again.

The exactly-once guarantee is about the committed, observable results within the supported transactional domain, not a promise that application instructions execute only once.

This is why external side effects deserve special attention.

9. The real difficulty: external systems

Consider a consumer that reads an order event and makes an HTTP request to a third-party shipping service.

The request asks the service to create a shipping label.

Kafka can coordinate transactions involving its own records and offsets, but the shipping service is not automatically a participant in that transaction.

Suppose the following happens:

1. The consumer sends the shipping request.

2. The shipping provider successfully creates a label.

3. The network connection fails before the acknowledgment reaches the consumer.

4. The consumer cannot determine whether the operation succeeded.

5. The consumer retries.

The shipping provider may now create a second label.

Even if the consumer uses Kafka transactions perfectly, Kafka cannot automatically undo the first external shipping request.

The system has crossed a transactional boundary.

Strategy A: Use an idempotency key at the external API

Some external services support idempotency keys.

Instead of treating every request as a new operation, the service uses a stable request identifier to recognize retries of the same logical action.

The application could derive its key from the order ID and operation type.

For example, the logical identifier might be based on order 1001 and its create-shipping-label operation.

If the provider enforces an appropriate idempotency contract, repeated requests using the same identifier can return the original outcome rather than create another resource.

However, this guarantee depends on the service’s documented behavior.

You must verify:

• Whether the API supports idempotency.

• How long keys remain valid.

• Whether the same key can safely be retried after a timeout.

• What happens if the same key is reused with a different payload.

• Whether idempotency records remain durable across provider failures.

Do not assume all APIs implement these rules identically.

Strategy B: Use a transactional inbox

Our SQLite example demonstrates a transactional inbox.

The consumer stores a record of each logical event it has successfully applied. The unique constraint prevents repeated application.

This works particularly well when the business mutation and deduplication record are stored in the same transactional database.

For example, an inventory consumer could atomically:

• Register an incoming order event.

• Reserve a quantity of inventory.

• Mark the event as processed.

If the same event arrives again, the database recognizes it and avoids reserving more inventory.

The limitation is that this does not automatically make an independent HTTP request atomic with the database transaction.

Strategy C: Use the transactional outbox pattern

The transactional outbox addresses a related problem on the producer side.

Suppose an order service needs to do two things:

1. Save an order in PostgreSQL.

2. Publish an OrderCreated event to Kafka.

If it writes to PostgreSQL first and crashes before producing the event, downstream services never learn about the order.

If it publishes first and the database transaction fails, downstream services may observe an order that doesn’t exist.

A transactional outbox changes the workflow.

The order service writes both the order and an outgoing event into its own database, within one database transaction.

A separate publisher subsequently reads pending outbox records and delivers them to Kafka.

The database now atomically guarantees that either both the order and its pending event exist or neither exists.

The outbox publisher may still produce the same logical event more than once if it crashes after Kafka acknowledges delivery but before the publisher records completion.

Therefore, a complete outbox-based architecture often combines:

• A transactional outbox on the publishing side.

• At-least-once event delivery.

• Stable event identifiers.

• A transactional inbox or equivalent idempotent processing on the receiving side.

This does not create one universal transaction spanning every system. Instead, it composes smaller reliable mechanisms into an application with well-defined business invariants.

Strategy D: Store processing progress with the business state

Another useful design is to store the last successfully applied Kafka offset in the same database transaction as the business output.

For each input partition, the application persists its processed position with its results.

On recovery, it can seek to the position recorded by that database.

This can coordinate business data and replay progress without relying on an atomic transaction between Kafka’s consumer offset storage and the external database.

The approach requires careful handling of partition ownership, offset sequencing, retention, and recovery, especially when scaling consumers.

Kafka’s own design documentation describes external offset storage as a way to coordinate processing results with consumer positions.

The key architectural lesson is that a database can enforce exactly-once business effects when it has enough information and transactional capabilities to identify and atomically record those effects.

10. The cherry on the cake: is exactly-once actually impossible?

One of the most interesting debates in distributed systems is whether exactly-once delivery is fundamentally impossible.

In June 2017, Confluent published a technical article by Neha Narkhede, Guozhang Wang, and colleagues titled Exactly-once Semantics Are Possible: Here’s How Kafka Does It.

The article directly addressed the widespread skepticism around exactly-once guarantees and explained how idempotent producers and transactions make exactly-once stream processing possible within Kafka’s defined scope.

The apparent contradiction becomes clearer when we distinguish the claims.

Claim 1: A distributed system cannot always know whether a remote side effect happened

Imagine sending a request to another machine and losing the connection before receiving a response.

The remote machine may have completed the operation. It may also have failed before doing anything.

From the sender’s immediate perspective, these outcomes can be indistinguishable.

If the sender retries, duplicate execution is possible.

If the sender never retries, the operation might never happen.

No choice of retry policy alone resolves both possibilities.

This is the practical basis for much of the skepticism about exactly-once delivery.

Claim 2: A system can ensure one committed observable result

Now consider a transactional database.

The application requests a modification identified by a unique key.

The database can enforce a uniqueness constraint and commit the result atomically.

The application might send the same request repeatedly, but the database can allow only one corresponding business effect.

The computation or network request may happen multiple times, while the persistent outcome is equivalent to one successful application.

Kafka transactions achieve a related result inside Kafka’s transactional domain.

The system coordinates the visibility of produced records with the advancement of consumed offsets.

Both claims can be true

The contradiction disappears when we stop treating “exactly once” as an unlimited promise.

A system can provide exactly-once processing semantics within a clearly specified transactional boundary without guaranteeing that every physical operation executes exactly one time.

Likewise, external systems can participate in stronger business-level guarantees when they provide suitable transactional or idempotency mechanisms.

What remains impossible to obtain automatically is universal exactly-once behavior across arbitrary external side effects that offer no way to coordinate, query, deduplicate, or reverse their outcomes.

The practical question is therefore not simply:

Does this platform support exactly-once processing?

It is:

Which observable effects are covered, what durable state establishes the guarantee, and what happens at every failure boundary?

That question is far more useful during architecture reviews.

11. Production engineering: the guarantees worth testing

A correct local demonstration is the beginning, not the end, of a reliable streaming design.

Production deployments introduce rebalances, multiple consumers, partial outages, schema changes, and operational failures that our one-record example intentionally avoids.

Handle rebalances carefully

Kafka can reassign partitions when consumers join or leave a group, fail, or lose membership.

A worker may still be executing business logic when its partition assignment changes.

If another worker takes over the same partition, both may temporarily have work associated with the same records.

Use the consumer client’s assignment and revocation lifecycle appropriately. Ensure that processing and offset management cannot race against ownership changes.

For Kafka transactions, producer fencing protects against obsolete transactional owners.

For external databases, durable uniqueness constraints and explicit ownership mechanisms may still be necessary.

Preserve ordering where it matters

Kafka maintains record order within an individual partition, not one global order across an entire topic.

If all updates for one order must be processed sequentially, a common strategy is to use the order ID as the message key so that the default partitioner directs records with that key to the same partition.

However, partition ordering alone does not guarantee correct business-state transitions if processing is parallelized incorrectly, reordered elsewhere, or replayed.

For example, a cancellation event should not accidentally overwrite a later state transition merely because a stale event is retried.

Consider version numbers, optimistic concurrency control, and explicit event-state transition rules where appropriate.

Avoid accidental offset advancement

Automatic offset management is convenient, but it can obscure failure windows.

A consumer should not acknowledge progress before the processing operations required by its delivery contract have completed.

When processing batches or multiple partitions concurrently, track successful progress per partition.

Do not commit beyond unfinished records simply because a later record has finished processing.

A single committed offset represents progress through an ordered sequence, not an arbitrary set of completed messages.

Treat malformed events separately from transient failures

A temporary database outage and a permanently malformed JSON event require different responses.

Repeatedly retrying malformed data may block a partition indefinitely.

Production applications usually need policies for:

• Transient failures with bounded backoff and retries.

• Records that violate an expected schema.

• Business validation errors.

• Dead-letter handling with enough context for investigation.

• Explicit decisions about whether and when failed records can be skipped.

A dead-letter topic does not automatically preserve exactly-once business semantics.

Moving a record to a dead-letter topic is itself a processing outcome that needs an explicit reliability contract.

Choose durability settings deliberately

For important data, configure broker replication and acknowledgment policies according to the required failure tolerance.

An idempotent producer and acks=all are valuable, but they do not protect against losing the only broker and its only copy of the log.

Production configurations should consider:

• Replication factor.

• Minimum in-sync replicas.

• Broker and storage failure domains.

• Topic retention.

• Transaction state durability.

• Consumer offset retention.

• Backup and disaster recovery requirements.

Kafka Streams documentation recommends replication factor 3 and an appropriately configured minimum in-sync replica count for durable exactly-once processing deployments.

Monitor more than consumer lag

Consumer lag is useful because it reveals how far a group is behind the topic’s current position.

But low lag does not prove correctness.

An application that commits offsets before completing business writes could appear completely caught up while silently missing operations.

Important operational signals include:

• Consumer lag and processing latency.

• Failed and retried processing attempts.

• Consumer rebalances and assignment changes.

• Offset commit failures.

• Transaction aborts and fencing errors.

• Duplicate event IDs encountered.

• Database constraint conflicts.

• Dead-letter volume and age.

• Differences between expected event counts and applied business operations.

Metrics should help answer whether the application is making progress and whether that progress represents correct results.

12. A practical failure-testing matrix

Our experiment reproduced one failure window. Real systems should test several.

The distinction between testing individual components and testing the complete failure sequence is important.

A unit test that verifies a deduplication function returns the expected result is useful.

A crash test that terminates a process immediately after its database commit exercises a much stronger property: whether durable business state and recovery behavior remain consistent under a realistic interruption.

Experiment further

Once you’ve completed the tutorial, try modifying the lab.

First, publish the same event twice and process both copies in safe mode. The first event should create the database record, while the second should encounter the existing idempotency key.

Next, move the simulated crash point before the SQLite transaction commits. Observe how the behavior changes.

Then consider extending the consumer to process batches or multiple partitions. You’ll need to track successful offsets more carefully and account for partial batch failures.

A particularly valuable exercise is to add a database operation that deliberately fails, verify that the event remains uncommitted, and show that a later retry can complete successfully.

Each experiment should have an explicit expected business invariant before you run it.

Reset the lab

To repeat the original experiment from scratch, stop and remove the Kafka container and delete the local SQLite file.

docker rm -f kafka-lab
rm -f orders.sqlite3

Then start Kafka again using the Docker command from the setup section.

This removes the local Kafka container state used in this example and resets the database, giving you fresh topics and consumer-group progress for another run.

Do not use this cleanup procedure against a container holding data you need to preserve.

13. Choosing the right strategy

There is no single setting that eliminates every duplicate-processing problem.

Instead, choose a mechanism that matches the boundary of the operation.

These mechanisms are complementary rather than mutually exclusive.

For example, a robust event-driven commerce system might use a transactional outbox in the order service, idempotent producers to deliver events, Kafka Streams transactions for internal transformations, and an idempotent inventory consumer backed by database constraints.

Each component protects a particular boundary.

Together, they can establish stronger end-to-end business guarantees than any single broker configuration can provide.

14. Final takeaways

Kafka’s reliability model becomes much easier to understand when you separate delivery, processing, and observable business outcomes.

An at-least-once consumer may process the same event repeatedly after a failure. That behavior is often necessary to prevent lost work.

Producer idempotence protects against duplicates caused by producer retries, but it does not prevent application-level duplicate events or consumer-side replay.

Kafka transactions can atomically coordinate produced Kafka records with consumed Kafka offsets. With the correct consumer isolation and transactional ownership, they provide exactly-once processing semantics inside Kafka’s supported scope.

External databases and APIs require additional cooperation. Unique event IDs, atomic database transactions, durable inboxes, outboxes, and external idempotency contracts are the mechanisms that bridge those boundaries.

Most importantly, exactly-once processing should be understood as a precise guarantee about committed observable effects, not a magical promise that every event physically executes only once.

Our experiment demonstrated the difference: Kafka replayed both order events after the simulated failures, but only the idempotent consumer preserved the intended one-row business result.

That’s the engineering goal worth aiming for.

Your next step: break the system on purpose

Run the producer and consumer experiment yourself, reproduce the duplicate database row, and then verify the idempotent fix.

After that, extend the example with a second consumer, introduce database failures, and test how your application recovers during partition reassignment.

For deeper exploration, use the current Apache Kafka 4.3 documentation and Confluent Python client API reference to build a transactional Kafka-to-Kafka pipeline.

The challenge: Define one business invariant—such as one inventory reservation per unique order event—and design a failure test that proves it survives crashes, retries, and redelivery.

Once you can explain and demonstrate why that invariant holds, you’re doing more than configuring message delivery. You’re engineering a reliable distributed system.