Migrate from HLRC to elasticsearch-java 8.x client

[Yeikel Santana (yeikel)]

Problem

The High Level REST Client (HLRC) is deprecated and was fully removed in Elasticsearch 9, blocking support for ES 9.x. The connector currently depends on elasticsearch-rest-high-level-client 7.x.

Fixes #847.

Breaking change: This migration drops support for Elasticsearch 7.x, which reached end of life in January 15, 2026. The minimum supported version is now Elasticsearch 8.x.

Solution

Replace HLRC with co.elastic.clients:elasticsearch-java 8.19.15.

Dependencies

• Removed elasticsearch-rest-high-level-client and elasticsearch (HLRC artifacts)
• Added co.elastic.clients:elasticsearch-java 8.19.15 and elasticsearch-rest-client 8.19.15
• Upgraded Jackson 2.15.2 → 2.18.6; added log4j-bom 2.25.4 to align transitive log4j versions
• Set Java 11 as the compiler release target (required by elasticsearch-java)

Core client

• BulkProcessor → BulkIngester<SinkRecordAndOffset>: offset state is carried as context on each bulk operation instead of being looked up in a side map
• Added DispatchingTransport — dispatches async callbacks to a dedicated executor, breaking a lock-ordering deadlock between BulkIngester's internal condition and the NIO thread pool
• Retry uses BackoffPolicy.exponentialBackoff

Data layer

• DataConverter.convertRecord(): returns BulkOperation instead of DocWriteRequest
• Mapping.buildMapping(): returns Map<String, Object> instead of XContentBuilder
• External versioning (VersionType.EXTERNAL) inlined — no direct equivalent in the new API

SSL startup scripts

• Removed chmod 777 ${ES_DIR} — script runs as elasticsearch user which cannot chmod its own install directory
• Replaced su - elasticsearch << EOF heredoc with exec /usr/local/bin/docker-entrypoint.sh eswrapper (ES 8.x process supervisor with correct signal forwarding)
• Replaced openssl pkcs12 with elasticsearch-certutil for PKCS12 client cert generation
• PATH update for the bundled JDK is now conditional on keytool not already being present

Does this solution apply anywhere else?

• yes
• no

If yes, where?

Test Strategy

Existing integration tests

Testing done:

• Unit tests
• Integration tests
• System tests
• Manual tests

Release Plan

Merge to master. No back port needed as this is a forward-only migration and it is not backwards compatible. We should consider releasing 16.x for this change

Notes

• The client support Java 8, but I think that this could be a good opportunity to bump it to a higher version like 11 or 17.

• The commit fc7f57d5 (.github/workflows/ci.yml): was added only to run the test suite via GitHub Actions since I don't have access to Confluent's Semaphore CI. This should be dropped before merging

• The commit 802f8780 (testcontainers 1.16.3 → 1.21.4): can be extracted into a standalone PR if preferred. It was required here because 1.16.3 fails against Docker Engine 29.x. which is what GitHub Action uses

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ yeikel
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[Yeikel Santana (yeikel)]

Can be removed before we merge. This was added to help me test end to end rather than locally only

[Yeikel Santana (yeikel)]

My goal with the migration was to producer the smaller diff possible. I can definitely fix this but the diff will be much larger. I think that it makes sense to take this in a subsequent change