Kafka 4.4 finally gives share groups a dead-letter queue

Share groups (Kafka Queues) have been production-ready since 4.2, but one question kept me from using them: where do bad messages go? A record that a consumer rejects, or that runs out of delivery attempts, moves to the Archived state. In practice, it’s gone. Kafka 4.4, released October 22, fixes that with KIP-1191.

The DLQ now lives in the broker

With classic consumer groups, I’ve always handled DLQs on the client. Spring Kafka’s DeadLetterPublishingRecoverer catches the exception and publishes to a .DLT topic. Every service wires it up itself, and every team names things differently.

Share groups are different because the broker tracks how many times each record was delivered (share.delivery.count.limit, default 5). So KIP-1191 lets the broker write the record to a DLQ itself when it’s rejected or exceeds the limit. You turn it on at three levels:

# 1. The target topic must opt in as a DLQ
kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --entity-type topics --entity-name dlq.orders \
  --add-config errors.deadletterqueue.group.enable=true

# 2. The group points at its DLQ, optionally copying the original record
kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --entity-type groups --entity-name order-workers \
  --add-config errors.deadletterqueue.topic.name=dlq.orders,errors.deadletterqueue.copy.record.enable=true

# 3. The cluster needs share.version=2
kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature share.version=2

A few defaults will trip you up. DLQ topic names must start with dlq. unless you change errors.deadletterqueue.topic.name.prefix. Auto-creating DLQ topics is off. And copy.record.enable defaults to false, so the DLQ gets only context headers (__dlq.errors.topic, __dlq.errors.offset, __dlq.errors.delivery.count, __dlq.errors.message) and no payload. If you want to replay, turn it on.

Read the “not atomic” part

The KIP says the DLQ write is not atomic with the share group state update, so in rare cases “more than one DLQ record could be written” for the same original record. If the write fails with a non-retriable error, the broker logs it and archives the record anyway.

I used to think of a DLQ as “guaranteed, exactly one copy”. It isn’t. Whatever replays from the DLQ has to be idempotent. Dedupe on topic + partition + offset from the headers.

Why I care

Most of the messaging incidents I’ve dealt with weren’t about throughput. They were one poison message swallowed silently, and someone asking days later where their order went. A broker-side DLQ means a new service can’t forget to configure one. The trade-off is that it becomes the cluster operator’s job, so agree on a naming convention early.

If you’re trying share groups, run kafka-features.sh describe first. If share.version isn’t at 2, your DLQ isn’t active, no matter what you set on the group.

Source: Apache Kafka 4.4.0 Release Announcement, KIP-1191: Dead-letter queues for share groups, Kafka 4.4 Group Configs