Apache Kafka Explained: Topics, Partitions, Consumer Groups and Delivery Guarantees

Apache Kafka is a distributed log. Programs called producers append records to it, and other programs called consumers read those records, each at its own pace. The records stay for a set time whether or not anyone has read them.

That one idea explains most of Kafka: why it handles high volume, why a slow consumer does not hurt anyone else, why you can replay old data, and why ordering has rules.

The terminal output in this post is real. I ran a single Kafka 4.3.1 broker locally and captured the commands and results, and the Java examples were compiled and run against it. Where a default value is quoted, I read it from that version. Older versions can differ.

1. Why Kafka exists

Suppose an order service must tell the payment, email and analytics services about every new order. Calling each one directly means the order service has to know all of them, wait for them, and cope when one is down or slow. Add more services and the number of connections grows fast.

Kafka sits in the middle. The order service writes one event to a topic and moves on. Each other service reads from the topic when it is ready. A new service can start reading without any change to the producer, and it can even start from the beginning to catch up.

Kafka was created at LinkedIn to move activity and log data between systems, and was open-sourced in 2011. It is now an Apache project.

2. The pieces

3. Topics and partitions

A topic is split into partitions. Each partition is an ordered log: records are only ever added to the end, and each gets the next offset.

Here is a real run. This creates a topic with 3 partitions on a one-broker cluster and shows its layout:

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3 --replication-factor 1
Created topic orders.
 
$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
Topic: orders   PartitionCount: 3   ReplicationFactor: 1   Configs: min.insync.replicas=1,segment.bytes=1073741824
    Topic: orders   Partition: 0   Leader: 1   Replicas: 1   Isr: 1
    Topic: orders   Partition: 1   Leader: 1   Replicas: 1   Isr: 1
    Topic: orders   Partition: 2   Leader: 1   Replicas: 1   Isr: 1

Now send ten records with keys, then read them back with their partition and offset. The consumer output below is grouped by partition for readability:

$ printf 'user-1:created\nuser-2:created\nuser-3:created\nuser-4:created\nuser-5:created\nuser-1:paid\nuser-2:paid\nuser-3:paid\nuser-1:shipped\nuser-4:paid\n' \
    | bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders \
      --reader-property parse.key=true --reader-property key.separator=:
 
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning \
    --formatter-property print.key=true --formatter-property print.partition=true --formatter-property print.offset=true
Partition:1   Offset:0   user-4   created
Partition:1   Offset:1   user-5   created
Partition:1   Offset:2   user-4   paid
Partition:2   Offset:0   user-1   created
Partition:2   Offset:1   user-2   created
Partition:2   Offset:2   user-3   created
Partition:2   Offset:3   user-1   paid
Partition:2   Offset:4   user-2   paid
Partition:2   Offset:5   user-3   paid
Partition:2   Offset:6   user-1   shipped

Three things to notice:

• Every record for one key landed in one partition. user-1 was created, paid and shipped, in that order, at offsets 0, 3 and 6 of partition 2.
• The keys are not spread evenly. Partition 0 received nothing. A hash does not promise a perfect split, especially with few keys.
• Offsets belong to a partition. Offset 0 exists in every partition. A record is identified by topic, partition and offset together.

The ordering rule

Kafka guarantees order within a partition, not across a topic. To keep all events for one entity in order, give them the same key. The producer hashes the key to choose the partition, so the same key always goes to the same partition.

The demo uses Kafka's real default partitioner (the murmur2 hash of the key, modulo the number of partitions). I checked it against the broker: it chose the same partition for all 40 key and partition-count combinations I tried.

What happens when you add partitions

You can add partitions to a topic but never remove them. After growing orders from 3 to 6 partitions, I sent the same keys again:

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic orders --partitions 6
$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic orders --partitions 2
Error while executing topic command : The topic orders currently has 6 partition(s); 2 would not be an increase.
 
user-1  created -> partition 2     again -> partition 2
user-2  created -> partition 2     again -> partition 2
user-3  created -> partition 2     again -> partition 5
user-4  created -> partition 1     again -> partition 1
user-5  created -> partition 1     again -> partition 4

user-3 and user-5 moved. Their old records stay where they were, so for those keys the order across the change is no longer guaranteed. Choose the partition count with some headroom from the start.

Records without a key

Without a key, there is no per-key ordering to keep, and the default producer fills a batch for one partition before moving to the next. In my test, 24 keyless records sent to a 4-partition topic all landed in partition 2. Do not rely on keyless records being evenly spread in small bursts.

4. Producers

A producer decides the partition, batches records, sends them to the partition's leader and handles retries. This Java example was compiled and run against the broker above:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");                  // wait for all in-sync replicas
props.put("enable.idempotence", "true");   // retries cannot create duplicates
 
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
  for (int i = 1; i <= 3; i++) {
    ProducerRecord<String, String> record =
        new ProducerRecord<>("orders", "user-" + i, "order-" + i);
    producer.send(record, (meta, error) -> {
      if (error != null) error.printStackTrace();
      else System.out.printf("sent to partition %d at offset %d%n", meta.partition(), meta.offset());
    });
  }
}   // close() flushes pending sends

Output:

sent to partition 2 at offset 15
sent to partition 2 at offset 16
sent to partition 5 at offset 5

send is asynchronous: it returns at once and the callback reports the result. Leaving the try block closes the producer, which waits for pending records.

The settings that matter most, with the defaults in the Kafka 4.3 client:

Older clients had different defaults: acks was 1 and idempotence was off before Kafka 3.0, so check the version you run.

5. Brokers and replication

A partition is stored on a broker, and a copy of it is stored on others. One copy is the leader. All reads and writes go through the leader. The others are followers that copy it. The followers that are fully caught up form the in-sync replicas (ISR).

If the leader's broker dies, one of the in-sync followers becomes the new leader. A common production setup is:

• replication.factor=3: three copies of each partition.
• acks=all: a write counts only when every in-sync replica has it.
• min.insync.replicas=2: refuse writes if fewer than two replicas are in sync.

With those three, the cluster keeps accepting writes with one broker down and never acknowledges a write that exists on only one machine. Note that the defaults on a fresh broker are default.replication.factor=1 and min.insync.replicas=1, so this has to be set on purpose.

Here is why acks matters. Step through the same crash with both settings:

6. Consumers and consumer groups

Consumers pull records from brokers; brokers never push. A consumer remembers its position in each partition as an offset.

Consumers that share a group.id form a consumer group, and Kafka splits the topic's partitions between them. The rule is simple: inside one group, each partition is read by exactly one consumer. Different groups are independent and each reads every record.

Here is the real thing. Two consumers joined a group called billing on the 6-partition orders topic, and I listed the group's members (some columns and the ends of the consumer IDs are trimmed here):

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing --members --verbose
GROUP    CONSUMER-ID                       #PARTITIONS   CURRENT-ASSIGNMENT
billing  console-consumer-4036...          3             orders:3,4,5
billing  console-consumer-1cfc...          3             orders:0,1,2

I then started a third consumer in the same group, and Kafka rebalanced:

billing  console-consumer-4036...          2             orders:2,3
billing  console-consumer-1cfc...          2             orders:0,1
billing  console-consumer-632e...          2             orders:4,5

The group's progress is tracked as committed offsets, stored in an internal topic called __consumer_offsets (50 partitions on this broker, compacted). The gap between the end of the log and the committed offset is the lag. This is the number to watch in production, because a growing lag means consumers are falling behind. After I stopped the consumers and sent more records:

$ bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing
GROUP    TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
billing  orders  0          0               0               0
billing  orders  1          4               6               2
billing  orders  2          9               15              6
billing  orders  3          0               2               2
billing  orders  4          1               3               2
billing  orders  5          1               5               4

The group kept its committed offsets while it had no members. When it returns, it resumes from there.

A consumer in Java, which was also compiled and run against this broker:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "inventory");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("enable.auto.commit", "false");     // commit only after the work is done
props.put("auto.offset.reset", "earliest");
 
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
  consumer.subscribe(List.of("orders"));
  while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
    for (ConsumerRecord<String, String> r : records) {
      System.out.printf("partition=%d offset=%d key=%s value=%s%n", r.partition(), r.offset(), r.key(), r.value());
    }
    consumer.commitSync();                    // at-least-once: commit after processing
  }
}

It read all 34 records in the topic and committed. After that the group showed zero lag on every partition.

Rebalancing and the timers that cause it

Whenever a consumer joins, leaves or is considered dead, the group rebalances and partitions move. Three timers decide "dead", with their Kafka 4.3 defaults:

• heartbeat.interval.ms = 3 seconds: how often a consumer says it is alive.
• session.timeout.ms = 45 seconds: no heartbeat for this long and it is removed.
• max.poll.interval.ms = 5 minutes: if your code takes longer than this between calls to poll, the consumer is removed even though it is running. A slow handler is the usual cause of endless rebalancing.

A new group with no committed offset starts at the end of the log by default (auto.offset.reset=latest). Use earliest to read the history.

7. Delivery guarantees

What a consumer can promise depends on one choice: when it commits the offset.

At-least-once is the usual choice, with a consumer that tolerates duplicates (for example, an update keyed by an ID rather than a blind insert).

Exactly once inside Kafka is real, but it has limits. The idempotent producer stops retries from writing a record twice. Transactions (transactional.id) let a program read, process and write to several partitions atomically, and consumers with isolation.level=read_committed skip uncommitted or aborted records. None of that covers an external system. If your consumer also sends an email or charges a card, make that action idempotent too.

8. Retention and compaction

Kafka does not delete a record when it is read. It deletes by policy:

• Time or size. The broker default is log.retention.hours=168, which is 7 days. Old log segments are deleted as a whole. A topic can override this with retention.ms or retention.bytes.
• Compaction. With cleanup.policy=compact, Kafka keeps at least the latest record for each key and discards older ones. This suits topics that hold the current state of something, such as a user's profile. The internal __consumer_offsets topic is compacted.

I created a compacted topic and sent five records, then waited for the cleaner. The records before and after:

Before compaction            After compaction
Offset:0  A  v1              Offset:3  B  v2
Offset:1  B  v1              Offset:4  A  v3
Offset:2  A  v2              Offset:5  C  v1
Offset:3  B  v2              Offset:6  D  v1
Offset:4  A  v3

(Offsets 5 and 6 were records I added to roll the log segment, because Kafka does not compact the segment that is currently being written.) The old values of A and B are gone, and the survivors keep their original offsets. Offsets are never renumbered.

To delete a key from a compacted topic, send a record with that key and a null value. This is called a tombstone.

9. Why Kafka is fast

Kafka's own documentation points to a few design choices:

• Sequential disk access. Records are appended to the end of a log file, which disks and operating systems handle very well.
• The operating system's page cache. Kafka leans on it rather than keeping a big cache of its own.
• Zero-copy transfer. The broker can send data from the file cache to the network without copying it through the application.
• Batching and compression. Producers send batches, and a whole batch can be compressed.
• Partitions. The work spreads across brokers and across consumers.

10. Kafka compared with a traditional queue

The line is blurrier than it used to be: RabbitMQ added a log-like feature called Streams in version 3.9. Treat the table as the classic difference, not a rule.

11. Where it is used

• Event-driven services. The order example from the start: one event, several independent reactions.
• Log and metrics collection. The original LinkedIn use: many servers write to Kafka, and storage and monitoring systems read from it.
• Change data capture. Tools such as Debezium read a database's transaction log and publish each row change to Kafka, so other systems can follow the database without querying it.
• Stream processing. Kafka Streams is a Java library for transforming and aggregating topics as they arrive.

12. Run it yourself

These are the steps I used. I ran them with Java 21 on Linux.

# 1. Download and unpack a release from https://downloads.apache.org/kafka/
tar -xzf kafka_2.13-4.3.1.tgz && cd kafka_2.13-4.3.1
 
# 2. Format the storage once, then start a single broker (KRaft, no ZooKeeper)
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.properties
bin/kafka-server-start.sh config/server.properties
 
# 3. In a second terminal: create a topic, produce, consume
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning

Type a few lines into the producer and watch them appear in the consumer. Newer releases use --reader-property for the producer and --formatter-property for the consumer; the older --property flag still works but prints a deprecation warning.

13. Interview questions

Does Kafka guarantee ordering? Within a partition only. Use the same key to put related records in the same partition.

How many consumers can read a topic in parallel? As many as it has partitions, within one consumer group. Extra consumers sit idle.

How do you avoid losing messages? On the producer, acks=all, replication factor 3, min.insync.replicas=2. On the consumer, commit offsets only after processing.

How do you avoid duplicates? Enable the idempotent producer, and make the consumer's work idempotent. Transactions give exactly-once between Kafka topics.

What is consumer lag, and why does it matter? The difference between the latest offset in a partition and the group's committed offset. Growing lag means consumers cannot keep up.

What triggers a rebalance? A consumer joining or leaving, a missed heartbeat past session.timeout.ms, or a gap between poll calls longer than max.poll.interval.ms.

Can you reduce the number of partitions? No. You can add partitions, but that changes which partition some keys map to.

Why is Kafka fast? Sequential writes, the page cache, zero-copy transfer, batching and compression, and parallelism across partitions.

Kafka or a queue? Choose Kafka when you want a retained, replayable stream that several independent consumers read. Choose a queue when you want task distribution with flexible routing and acknowledgement per message.

14. Common mistakes

• Too few partitions at the start. You can add later, but keys move.
• No key when order matters. Without a key there is no per-key ordering.
• Relying on auto-commit. With enable.auto.commit=true (the default), offsets are committed every 5 seconds regardless of whether your processing finished.
• Slow handlers. Work that takes longer than max.poll.interval.ms causes rebalances.
• Large messages. The default maximum message size is about 1 MB (message.max.bytes=1048588). Store big files elsewhere and send a reference.
• Treating Kafka as a database for arbitrary queries. It is excellent for ordered streams and replay, not for ad hoc lookups.
• Production with a replication factor of 1. The broker default is 1, so set it on purpose.

Sources and further reading

• The Apache Kafka documentation, especially the sections on design, the producer and consumer configuration, and replication. The default values quoted in this post come from the Kafka 4.3.1 clients and broker.
• Neha Narkhede, Gwen Shapira and Todd Palino, Kafka: The Definitive Guide.
• Jay Kreps, "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013).
• KIP-98, exactly-once delivery and transactional messaging, and KIP-500, replacing ZooKeeper with a self-managed metadata quorum, on the Apache Kafka wiki.
• Martin Kleppmann, Designing Data-Intensive Applications, the chapter on stream processing.