Convert org.apache.kafka.connect.data.Date to epoch millis before sending to Elasticsearch

[Aleksander Lech (aleksanderlech)]

Problem

Lets consider the following kafka schema:

{
      "name": "dateOfBirth",
      "type": [ "null", {"type": "int", "logicalType": "date"}],
      "default": null
}

And the target index field:

  "dateOfBirth": {
      "type": "date"
    }

Currently the date type (org.apache.kafka.connect.data.Date) is being converted to int (epoch days) and ends up in Elasticsearch being interpreted as epoch millis turning it into some datetime long in the past.

Elasticsearch does not suport epoch days type for date fields.

Solution

Assuming the field is properly indexed as date we should pass some of default supported formats like epoch_millis.

Does this solution apply anywhere else?

• yes
• no

If yes, where?

Test Strategy

Testing done:

• Unit tests
• Integration tests
• System tests
• [x ] Manual tests

Release Plan

Bugfix, it might be have some incompatibles with existing user workarounds though.

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ aleksanderlech
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[Aleksander Lech (aleksanderlech)]

Hello any chance of merging?