ERROR Encountered metadata publishing fault: Error deleting stray partitions during startup

[grindlife]

Self-Hosted Version

25.6.2

CPU Architecture

x86_64

Docker Version

25.0.8

Docker Compose Version

2.38.2

Machine Specification

• My system meets the minimum system requirements of Sentry

Steps to Reproduce

1. Run the install.sh script for version 23.11.2 to perform the installation.
2. Upgrade to version 24.8.0.
3. Run docker compose up -d.

Expected Result

No errors should occur during Kafka container startup.

Actual Result

When starting Sentry, the Kafka container outputs the following log:

ERROR Encountered metadata publishing fault: Error deleting stray partitions during startup (org.apache.kafka.server.fault.LoggingFaultHandler)
java.lang.RuntimeException: The log dir Log(dir=/var/lib/kafka/data/snuba-dead-letter-metrics-distributions-0, topic=snuba-dead-letter-metrics-distributions, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) does not have a topic ID, which is not allowed when running in KRaft mode.
	at kafka.server.metadata.BrokerMetadataPublisher$.$anonfun$findStrayPartitions$2(BrokerMetadataPublisher.scala:76)
	at scala.Option.getOrElse(Option.scala:201)
	at kafka.server.metadata.BrokerMetadataPublisher$.$anonfun$findStrayPartitions$1(BrokerMetadataPublisher.scala:75)
	at scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:118)
	at scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:105)
	at scala.collection.mutable.ArrayBuffer.flatMap(ArrayBuffer.scala:43)
	at kafka.server.metadata.BrokerMetadataPublisher$.findStrayPartitions(BrokerMetadataPublisher.scala:73)
	at kafka.server.metadata.BrokerMetadataPublisher.finishInitializingReplicaManager(BrokerMetadataPublisher.scala:353)
	at kafka.server.metadata.BrokerMetadataPublisher.onMetadataUpdate(BrokerMetadataPublisher.scala:246)
	at org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:309)
	at org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266)
	at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
	at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
	at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
	at java.base/java.lang.Thread.run(Thread.java:829)

• I started using Sentry Self-hosted from version 23.11.2 and upgraded sequentially to 24.8.0 -> 24.12.1 -> 25.5.1 -> 26.6.2, and now I am on the latest version. However, even after upgrading to 26.6.2, the error is still being output.
• Is the snuba-dead-letter-metrics-distributions topic no longer used in recent versions of Sentry?
• Looking at the Snuba source code, I noticed that in 23.11.2, the topic snuba-dead-letter-metrics-distributions was defined, but it was no longer defined in 24.2.0.
• Upon inspection inside the Kafka container, I found that snuba-dead-letter-metrics-distributions-0 exists.
• Should I manually delete /var/lib/kafka/data/snuba-dead-letter-metrics-distributions-0 in the Kafka container to prevent further errors during startup?
• Could you provide guidance on how to resolve this issue?

Thank you for your support!

Event ID

No response

[aldy505]

If you are fine with losing the data from the last 24 hours, you can prune and re-create yoursentry-kafka and sentry-kafka-log (see here) volumes: docker compose down && docker volume rm sentry-kafka && docker compose create sentry-kafka && ./install.sh && docker compose up -d. This is a rather a quick fix, though.

Should I manually delete /var/lib/kafka/data/snuba-dead-letter-metrics-distributions-0 in the Kafka container to prevent further errors during startup?

I believe this should suffice.

[grindlife]

If you are fine with losing the data from the last 24 hours, you can prune and re-create yoursentry-kafka and sentry-kafka-log (see here) volumes: docker compose down && docker volume rm sentry-kafka && docker compose create sentry-kafka && ./install.sh && docker compose up -d. This is a rather a quick fix, though.

Should I manually delete /var/lib/kafka/data/snuba-dead-letter-metrics-distributions-0 in the Kafka container to prevent further errors during startup?

I believe this should suffice.

Thank you for your prompt response and detailed explanation. I appreciate the clarification and the suggested approaches. This was very helpful!

[aldy505]

Did it solve the problem?