[2024-05-30 10:25:31,403] ERROR [hdfs3_sink-test_v4|task-0] WorkerSinkTask{id=hdfs3_sink-test_v4-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:237) java.lang.NullPointerException: Cannot invoke "io.confluent.connect.hdfs3.DataWriter.open(java.util.Collection)" because "this.hdfsWriter" is null

[NhatDuy11]

Hi every one, i am using connector hdfs-s3 sink connectors kafka confluent sink to hdfs , my config sink :
{
"name": "hdfs3_sink-test_v4",
"config": {
"key.converter.schemas.enabled": "true",
"value.converter.schemas.enabled": "true",
"schema.enable": "true",
"value.converter.schema.registry.url": "http://kafka01:8081,http://kafka02:8081",
"key.converter.schema.registry.url": "http://kafka01:8081,http://kafka02:8081",
"name": "hdfs3_sink-test_v4",
"connector.class": "io.confluent.connect.hdfs3.Hdfs3SinkConnector",
"tasks.max": "1",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"errors.tolerance": "all",
"errors.log.enable": "true",
"errors.log.include.messages": "true",
"topics": "dwdb.GGADMIN.TEST_DECIMAL",
"errors.deadletterqueue.topic.name": "fail_topic",
"errors.deadletterqueue.topic.replication.factor": "3",
"errors.deadletterqueue.context.headers.enable": "true",
"hdfs.url": "",
"hadoop.conf.dir": "/opt/cloudera/parcels/CDH/lib/hadoop",
"logs.dir": "/opt/hdfs/log_table",
"flush.size": "0",
"enhanced.avro.schema.support": "true",
"connect.meta.data": "true",
"schema.compatibility": "NONE",
"topics.dir": "/opt/hdfs/test_table",
"store.url": "hdfs://*************:9000",
"partitioner.class": "io.confluent.connect.storage.partitioner.DefaultPartitioner",
"timezone": "Asia/Ho_Chi_Minh",
"storage.class": "io.confluent.connect.hdfs3.storage.HdfsStorage",
"format.class": "io.confluent.connect.hdfs3.parquet.ParquetFormat",
"confluent.topic.bootstrap.servers": "kafka01:9092, kafka02:9092, kafka03:9092",
"confluent.topic.replication.factor": "3"
}
}
i am getting error :

[2024-05-30 10:25:31,403] WARN [hdfs3_sink-test_v4|task-0] WorkerSinkTask{id=hdfs3_sink-test_v4-0} Offset commit failed during close (org.apache.kafka.connect.runtime.WorkerSinkTask:418)
[2024-05-30 10:25:31,403] ERROR [hdfs3_sink-test_v4|task-0] WorkerSinkTask{id=hdfs3_sink-test_v4-0} Commit of offsets threw an unexpected exception for sequence number 1: null (org.apache.kafka.connect.runtime.WorkerSinkTask:276)
java.lang.NullPointerException: Cannot invoke "io.confluent.connect.hdfs3.DataWriter.getCommittedOffsets()" because "this.hdfsWriter" is null
at io.confluent.connect.hdfs3.Hdfs3SinkTask.preCommit(Hdfs3SinkTask.java:123)
at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:415)
at org.apache.kafka.connect.runtime.WorkerSinkTask.closePartitions(WorkerSinkTask.java:663)
at org.apache.kafka.connect.runtime.WorkerSinkTask.closeAllPartitions(WorkerSinkTask.java:658)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:208)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:229)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:284)
at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
[2024-05-30 10:25:31,403] ERROR [hdfs3_sink-test_v4|task-0] WorkerSinkTask{id=hdfs3_sink-test_v4-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:237)
java.lang.NullPointerException: Cannot invoke "io.confluent.connect.hdfs3.DataWriter.open(java.util.Collection)" because "this.hdfsWriter" is null

Does anyone have an opinion on this issue?
Thank you everyone for looking at this issue!

[nishantbahri]

Did you find any resolution to this?