🔖 Bug Report — KafkaJS with Kafka 4.0.0 (KRaft) group coordinator issue

[adham-2002]

Describe the bug
When running a KafkaJS consumer against a Kafka 4.0.0 (KRaft mode) broker, the consumer fails to join the group and repeatedly logs the error:

[Connection] Response GroupCoordinator(key: 10, version: 2) broker: localhost:29092 clientId: my-consumer error: The group coordinator is not available

This issue occurs when trying to auto-create the __consumer_offsets topic, which fails in KRaft mode, causing the consumer group coordination to be unavailable.

---

To Reproduce

Steps to reproduce the behavior:

1. Set up a Kafka broker using apache/kafka:4.0.0 Docker image in KRaft mode
2. Run a simple producer sending messages to a topic using KafkaJS
3. Run a consumer with groupId using KafkaJS subscribing to that topic
4. Observe the error in the consumer logs

---

Expected behavior
Consumer should successfully connect to the broker, join the group, and start consuming messages.

---

Observed behavior
Consumer keeps retrying to connect to the group coordinator and logs the following error repeatedly:

[Connection] Response GroupCoordinator(key: 10, version: 2) broker: localhost:29092 clientId: my-consumer error: The group coordinator is not available

And on the broker side:

Sent auto-creation request for Set(__consumer_offsets) to the active controller.
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = -720899)

---

Environment

• OS: Ubuntu 22.04 (VMware)
• KafkaJS version: 2.2.4
• Kafka version: 4.0.0 (KRaft mode)
• NodeJS version: 20.11.0
• Docker: 25.0.3

---

Additional context
This issue seems to be related to the recent changes in Kafka 4.x KRaft mode group management and how KafkaJS interacts with the GroupCoordinator API.

Temporary workaround:
Downgrading Kafka to 3.6.1 resolves the issue as group management is compatible in that version.

---

[DanielCyran]

Is anyone still developing KafkaJS?

[adham-2002]

Is anyone still developing KafkaJS?

you mean it not used ? and no one interested about it ?

[alexd6631]

Hello @adham-2002

I am also using KafkaJS at work and I was intrigued by your issue, since it could be a road blocker if at some point we migrate to Kafka 4 clusters.

kafka-test.zip

I made a simple test project, with docker-compose using confluentinc/cp-kafka:8.0.0 (which should embed Kafka 4), but was unable to reproduce the issue. While I had a few transient errors when the topic was just created, I am able to produce and consume messages. If you want to run the project, make sure to have a recent Node version installed, since it is using erasure typescript syntax to run natively.

Do you see what could differ in your setup? Did you use a real Kafka cluster or a dockerized instance?

Edit, just re-read your message, you are using the apache/kafka:4.0.0 image directly, on my side I am using cp-kafka:8.0.0 and it has been tricky to find the good env variables to configure the image. In the docker-compose you can see the variables I have been using, I am also including a redpanda console to view the topics. Can you also configure a red panda console to your local kafka, try to create and populate a topic, to make sure the problem is indeed related to kafkajs, and not the setup itself ?

BTW, there is a new official Confluent Kafka client for Node, whose design is heavily inspired by the KafkaJS API, and should get long-term maintenance from Confluent. You should consider migrating to it at some point, since the project seems no longer maintained. On the good side, it seems the project was battle-tested, and I had no issue in production yet with the latest 2.2.4 release. In theory, the consumer / producer API should not change despite the migration to Kraft, so in theory the client should work as is.