Migrate from HLRC to elasticsearch-java 8.x client

[Yeikel Santana (yeikel)]

Problem

The High Level REST Client (HLRC) is deprecated and was fully removed in Elasticsearch 9, blocking support for ES 9.x. The connector currently depends on elasticsearch-rest-high-level-client 7.x.

Fixes #847.

Breaking change: This migration drops support for Elasticsearch 7.x, which reached end of life on January 15, 2026. The minimum supported version is now Elasticsearch 8.x.

Solution

Replace HLRC with co.elastic.clients:elasticsearch-java 8.19.15.

Dependencies

• Removed elasticsearch-rest-high-level-client and elasticsearch (HLRC artifacts)
• Added co.elastic.clients:elasticsearch-java 8.19.15 and elasticsearch-rest-client 8.19.15
• Upgraded Jackson 2.15.2 → 2.18.6; added log4j-bom 2.25.4 to align transitive log4j versions pulled in by the new client
• Set Java 11 as the compiler release target (required by elasticsearch-java)
• Bumped Testcontainers 1.16.3 → 1.21.4 to fix Docker Engine 29.x incompatibility

Core client

• BulkProcessor → BulkIngester<SinkRecordAndOffset>: offset state is carried as context on each bulk operation instead of being looked up in a side map
• Added DispatchingTransport — dispatches async callbacks to a dedicated executor, breaking a lock-ordering deadlock between BulkIngester's internal condition and the NIO thread pool. Uses a separate retryScheduler thread pool to prevent a second deadlock where retry tasks queued on bulkScheduler would block behind BulkIngester's own flush tasks
• Removed BulkIngester.backoffPolicy to give DispatchingTransport sole ownership of the retry budget; previously both layers retried independently, causing up to (maxRetries+1)² HTTP requests per document
• Removed blocking close() from throwIfFailed() — the Connect framework calls task.stop() → client.close() immediately after put() throws, so the defensive close was redundant and caused up to 3-minute delays before a task transitioned to FAILED

Data layer

• DataConverter.convertRecord(): returns BulkOperation instead of DocWriteRequest
• Mapping.buildMapping(): returns Map<String, Object> instead of XContentBuilder
• External versioning (VersionType.EXTERNAL) inlined per operation — no single-call equivalent in the new builder API

SSL startup scripts

• Removed chmod 777 ${ES_DIR} — script runs as elasticsearch user which cannot chmod its own install directory
• Replaced su - elasticsearch << EOF heredoc with exec /usr/local/bin/docker-entrypoint.sh eswrapper for correct signal forwarding
• Replaced openssl pkcs12 with elasticsearch-certutil for PKCS12 client cert generation
• PATH update for the bundled JDK is now conditional on keytool not already being present

Does this solution apply anywhere else?

• yes
• no

Test Strategy

• Unit tests
• Integration tests (full suite via Testcontainers against a real ES 8.x instance)
• System tests
• Manual tests

Release Plan

Merge to master. No backport needed this is a forward-only migration and is not backwards compatible with ES 7.x. A new major version (16.x) should be considered for this release.

Notes

• Java target: the client supports Java 8 at runtime, but this could be a good opportunity to formally bump the minimum JVM requirement to 11 or 17 in a follow-up
• The ci: add GitHub Actions workflow commit should be dropped before merging it was added only to run the test suite via GitHub Actions since Confluent's Semaphore CI is not accessible from this fork
• The build: upgrade testcontainers commit can be extracted into a standalone PR if preferred

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ yeikel
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[Yeikel Santana (yeikel)]

Can be removed before we merge. This was added to help me test end to end rather than locally only

[Yeikel Santana (yeikel)]

My goal with the migration was to producer the smaller diff possible. I can definitely fix this but the diff will be much larger. I think that it makes sense to take this in a subsequent change

[Yeikel Santana (yeikel)]

Jainam Jain (@jjain1259) Is there anything you can do to help this one move forward?