Summary
If the one-shot Elasticsearch version probe in ElasticsearchClient's constructor fails transiently at task start (e.g. a brief Connection refused while a pod/node is coming up if running in kubernetes), the connector silently falls back to UNKNOWN_VERSION_TAG and never enables ES 8 API compatibility mode — for the entire lifetime of the task, with no re-check.
Against an ES 8.x server this leaves the bundled RestHighLevelClient 7.17 unable to parse any bulk response that contains successful items (they lack _type without the compatible-with=7 Accept header; the parser hits Objects.requireNonNull in DocWriteResponse — same NPE signature as #620). The task does not fail — it degrades into a pathological but "working" loop, which makes this very hard to diagnose in production.
Verified on 15.0.0; the relevant code is unchanged in master/v15.1.2.
Environment
- Connector: kafka-connect-elasticsearch 15.0.0 (from Confluent Hub zip)
- Elasticsearch: 8.19.12 (ECK)
- Kafka Connect: 4.0.0 (Strimzi), Java 17 (Temurin/Red Hat 17.0.15)
- Connector config: default
write.method=insert, key.ignore=false (⇒ external versioning by record offset), behavior defaults, errors.retry.timeout=-1
What happens
- At task start,
ElasticsearchClient constructor calls getServerVersion(). The probe got Connection refused (ES service unreachable for a few seconds on a freshly provisioned node):
WARN [task-thread-elastic-sink-0] ElasticsearchClient:191 - Failed to get ES server version
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused
at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
...
at org.elasticsearch.client.RestHighLevelClient.info(RestHighLevelClient.java:969)
at io.confluent.connect.elasticsearch.ElasticsearchClient.getServerVersion(ElasticsearchClient.java:185)
at io.confluent.connect.elasticsearch.ElasticsearchClient.<init>(ElasticsearchClient.java:146)
at io.confluent.connect.elasticsearch.ElasticsearchSinkTask.start(ElasticsearchSinkTask.java:82)
-
getServerVersion() swallows the exception and returns UNKNOWN_VERSION_TAG, so shouldSetCompatibilityToES8() returns false and setApiCompatibilityMode(true) is never called. There is no retry and no later re-check.
-
From then on, every bulk request enters this loop (captured with the low-level REST client tracer logger):
- The bulk request is sent without compatibility headers. ES 8 indexes the documents and returns
HTTP 200 with items like {"_index":...,"_id":...,"_version":...,"result":"created",...,"status":201} — no _type field.
- RHLC 7.17 throws a bare
java.lang.NullPointerException parsing the successful items. RetryUtil logs:
WARN RetryUtil - Failed to execute bulk request due to java.lang.NullPointerException. Retrying attempt (1/11) after backoff of 7313 ms
- After a random 0–10 s backoff, the retry re-sends the same batch. Since the first attempt actually indexed the documents, every item now returns
409 version_conflict_engine_exception (external versioning). Failure items don't go through the DocWriteResponse constructor, so they parse fine, the connector treats external-version conflicts as benign, and the batch is reported successful.
-
Net effect: the task stays RUNNING, nothing reaches the DLQ or ERROR level, but every batch costs two round-trips plus an average ~5 s backoff. Throughput collapsed to ~70 rec/s (vs ~370 rec/s for the identical setup where the startup probe succeeded), consumer lag grew unboundedly, and every document is written twice.
The asymmetry (success ⇒ NPE, conflict ⇒ parsed OK) plus the always-(1/11) retry counter make this extremely confusing to operators: the connector reports no errors while being ~5x too slow, and the NPE has no message or stack trace at WARN level.
Reproduction
- Run the connector against ES 8.x, but make ES unreachable for the few seconds during which the sink task starts (e.g. a NetworkPolicy/iptables drop, or start Connect before ES on a fresh cluster). Observe
WARN Failed to get ES server version.
- Restore connectivity. Produce records with keys (
key.ignore=false).
- Every bulk now logs
Failed to execute bulk request due to java.lang.NullPointerException. Retrying attempt (1/11), while documents do get indexed (twice). A task restart with ES reachable fixes it, which confirms the startup probe is the trigger.
Suggested fix
Any (or several) of:
- Retry
getServerVersion() with the connector's existing max.retries/retry.backoff.ms policy instead of a single attempt;
- Fail the task (
ConnectException) when the version cannot be determined, so the framework/operator restart policies handle it instead of starting a client in a silently broken mode;
- Re-probe the version lazily on the first successful connection, rather than only in the constructor;
- Or expose a config to force compatibility mode explicitly (there is currently no config that reaches
setApiCompatibilityMode).
Workaround
Restart the connector task while ES is reachable (POST /connectors/<name>/tasks/<id>/restart) — the probe reruns and compatibility mode engages (INFO Staring client in ES 8 compatibility mode).
Related issues (not duplicates)
Summary
If the one-shot Elasticsearch version probe in
ElasticsearchClient's constructor fails transiently at task start (e.g. a briefConnection refusedwhile a pod/node is coming up if running in kubernetes), the connector silently falls back toUNKNOWN_VERSION_TAGand never enables ES 8 API compatibility mode — for the entire lifetime of the task, with no re-check.Against an ES 8.x server this leaves the bundled RestHighLevelClient 7.17 unable to parse any bulk response that contains successful items (they lack
_typewithout thecompatible-with=7Accept header; the parser hitsObjects.requireNonNullinDocWriteResponse— same NPE signature as #620). The task does not fail — it degrades into a pathological but "working" loop, which makes this very hard to diagnose in production.Verified on 15.0.0; the relevant code is unchanged in
master/v15.1.2.Environment
write.method=insert,key.ignore=false(⇒ external versioning by record offset),behaviordefaults,errors.retry.timeout=-1What happens
ElasticsearchClientconstructor callsgetServerVersion(). The probe gotConnection refused(ES service unreachable for a few seconds on a freshly provisioned node):getServerVersion()swallows the exception and returnsUNKNOWN_VERSION_TAG, soshouldSetCompatibilityToES8()returnsfalseandsetApiCompatibilityMode(true)is never called. There is no retry and no later re-check.From then on, every bulk request enters this loop (captured with the low-level REST client
tracerlogger):HTTP 200with items like{"_index":...,"_id":...,"_version":...,"result":"created",...,"status":201}— no_typefield.java.lang.NullPointerExceptionparsing the successful items.RetryUtillogs:WARN RetryUtil - Failed to execute bulk request due to java.lang.NullPointerException. Retrying attempt (1/11) after backoff of 7313 ms409 version_conflict_engine_exception(external versioning). Failure items don't go through theDocWriteResponseconstructor, so they parse fine, the connector treats external-version conflicts as benign, and the batch is reported successful.Net effect: the task stays
RUNNING, nothing reaches the DLQ orERRORlevel, but every batch costs two round-trips plus an average ~5 s backoff. Throughput collapsed to ~70 rec/s (vs ~370 rec/s for the identical setup where the startup probe succeeded), consumer lag grew unboundedly, and every document is written twice.The asymmetry (success ⇒ NPE, conflict ⇒ parsed OK) plus the always-
(1/11)retry counter make this extremely confusing to operators: the connector reports no errors while being ~5x too slow, and the NPE has no message or stack trace at WARN level.Reproduction
WARN Failed to get ES server version.key.ignore=false).Failed to execute bulk request due to java.lang.NullPointerException. Retrying attempt (1/11), while documents do get indexed (twice). A task restart with ES reachable fixes it, which confirms the startup probe is the trigger.Suggested fix
Any (or several) of:
getServerVersion()with the connector's existingmax.retries/retry.backoff.mspolicy instead of a single attempt;ConnectException) when the version cannot be determined, so the framework/operator restart policies handle it instead of starting a client in a silently broken mode;setApiCompatibilityMode).Workaround
Restart the connector task while ES is reachable (
POST /connectors/<name>/tasks/<id>/restart) — the probe reruns and compatibility mode engages (INFO Staring client in ES 8 compatibility mode).Related issues (not duplicates)
Objects.requireNonNullinDocWriteResponse) but from the pre-14.x era when compatibility mode didn't exist at all; this report is about the 14.x/15.x compatibility mode failing to arm when the startup probe fails transiently.Failed to execute bulk request; this comment shows exactly this failure mode in the wild (NPE onattempt (1/6)immediately followed by version-conflict warnings on the retried batch), undiagnosed.