Kafka Consumer Lag Due to File Descriptor Exhaustion

intermediate

linuxkafka

18 min

100%

Kafka Cluster Service Topology

⚙️

Confluent Control Center

service

⚙️

Order Service (Producer)

service

📨

orders.placed

queue

⚙️

kafka-broker-1

service

⚙️

kafka-broker-2

service

⚙️

kafka-broker-3

service

⚙️

order-processing-cg (Consumer)

service

🗄️

ZooKeeper Ensemble

database

healthy

degraded

failed

unreachable

Kafka CLI

kafka> 

0/1 commands

context

PagerDuty fires a P2 alert at 14:32 UTC. Consumer lag on the orders.placed topic has crossed the 500,000 message threshold. Confluent Control Center shows the order-processing-cg consumer group in an 'Empty' state — no active consumers registered.

explanation

The order-processing consumer group is critical path for the checkout pipeline. Every message represents a placed order that needs fulfillment processing. At current production rate of ~2,000 msg/sec, 500K lag means roughly 4 minutes of unprocessed orders.

Engineer Thought

“Okay, this is bad. 2.3 million messages behind across 6 partitions and the consumer group is showing Empty. No active consumers at all. This isn't a slow consumer problem — something killed the entire group.”

Concepts

Consumer Group State: Empty

A Kafka consumer group enters the 'Empty' state when all consumers have left the group. This means no consumer instances are actively polling for records, causing lag to accumulate indefinitely.

Instructor Notes (1)

mindset+45s

When you see a consumer group in 'Empty' state with high lag, resist the urge to restart consumers immediately. The root cause is almost never the consumers themselves — something upstream is preventing them from connecting or being assigned partitions.