Add Elasticsearch 9.x support by mattmenefee · Pull Request #1009 · toptal/chewy

mattmenefee · 2026-03-17T17:53:19Z

Note: This PR depends on #1008 (CI and test infrastructure improvements) and #1010 (migration guide and doc fixes) which should be merged first.

Related: #1028 removes the obsolete _type field (mock helpers, EVERFIELDS, Index::Wrapper accessors).

Summary

Widen elasticsearch gem dependency from < 9.0 to < 10.0 to support ES 9.x clusters
Restructure scroll API calls to nest scroll_id under body:, matching the elasticsearch-ruby 9.x client interface
Use .body accessor on cat.indices response for ES 9.x gem compatibility
Pin elasticsearch gem to ~> 8.14 in CI for ES 8.x matrix entries (the 9.x gem sends a compatible-with=9 header that ES 8.x rejects)
Add ES version matrix to CI: 9.4.1 default with three 8.19.3 spot-check entries (Ruby 3.3/3.4/4.0)
Add a Chewy 8→9 migration guide section with code examples, gem pinning guidance, and breaking-change notes
Add a Chewy 9.0.x (ES 8.x + 9.x) row to the README compatibility table

Breaking changes

The search_query.chewy notification payload for scroll requests now nests scroll_id under body: (i.e. {scroll: '1m', body: {scroll_id: '...'}} instead of {scroll: '1m', scroll_id: '...'}). Update any application code that subscribes to scroll notification payloads.

Test plan

Verify all existing specs pass against ES 8.x
Verify all existing specs pass against ES 9.x
Verify scroll pagination works end-to-end on both ES versions
Verify search_query.chewy notification payload matches new format

mattmenefee · 2026-03-17T17:54:41Z

Note: This PR depends on #1008 (CI and test infrastructure improvements) and #1010 (migration guide and doc fixes) which should be merged first. Once those are merged, I'll rebase this branch to resolve any conflicts.

bbatsov

Thanks for tackling ES 9 support! We'll review this properly after #1008 and #1010 have been updated and merged, since this builds on top of both.

One thing I'd suggest splitting out now: the Sidekiq testing API changes (require 'sidekiq/testing' -> Sidekiq.testing!(:fake), Sidekiq::Testing.inline! -> Sidekiq.testing!(:inline)) are unrelated to ES 9 and would be easier to review and merge as a separate PR.

AlfonsoUceda · 2026-03-25T14:39:18Z

@mattmenefee I will take a look at this PR. Currently we need to upgrade to ES 8 so would be nice to test it myself when doing the migration.

mattmenefee · 2026-03-25T14:40:18Z

@AlfonsoUceda no problem and thanks so much! Let me know if any changes need to be made.

mattmenefee · 2026-06-01T23:04:24Z

      matrix:
        ruby: [ '3.2', '3.3', '3.4', '4.0' ]
        gemfile: [rails.7.2.activerecord, rails.8.0.activerecord, rails.8.1.activerecord]
-    name: ${{ matrix.ruby }}-${{ matrix.gemfile }}


@AlfonsoUceda Would it be better / cleaner to use something like appraisal to test against both Elasticsearch v8 and v9?

mattmenefee · 2026-06-03T18:35:58Z

@AlfonsoUceda I've been rebasing this onto master periodically as new commits have landed. Do you want me to keep doing that until this PR is merged?

I've mainly been doing this because I'm running this branch in Production, and picking up the new changes/improvements on master in addition to the upgrade to ES v9 has been convenient.

Elasticsearch 9.x introduced breaking changes to the elasticsearch-ruby client gem that prevent Chewy from working with ES 9 clusters: 1. The scroll API now requires scroll_id nested under a body: key ({scroll: '1m', body: {scroll_id: '...'}}) rather than as a top-level parameter. Without this, scroll requests raise an ArgumentError in elasticsearch-ruby 9.x. 2. API responses are now wrapped in Elasticsearch::API::Response objects instead of returning raw hashes/arrays. Code that calls .map or other Enumerable methods directly on responses (e.g. cat.indices) must now call .body first to access the underlying data. 3. The elasticsearch Ruby gem at version 9.x sends a compatible-with=9 header that ES 8.x servers reject. CI matrix entries that test against ES 8.x must pin the gem to ~> 8.14. Changes: - Widen elasticsearch gem dependency from < 9.0 to < 10.0 - Restructure perform_scroll to nest scroll_id under body:, which is compatible with both elasticsearch-ruby 8.x and 9.x - Access .body on cat.indices response in the drop_indices test helper - Add elasticsearch version matrix to CI: 9.4.3 default with three 8.19.3 spot-check entries (Ruby 3.3/3.4/4.0), using dynamic service container images - Pin elasticsearch gem to ~> 8.14 in CI for ES 8.x matrix entries - Add scroll body format assertion in scrolling specs - Add a Chewy 8->9 migration guide section with code examples, gem pinning guidance, and breaking-change notes - Add Chewy 9.0.x (ES 8.x + 9.x) row to the README compatibility table

mattmenefee force-pushed the elasticsearch-9-support branch from e9bc5d7 to dbd4a3a Compare March 17, 2026 17:53

mattmenefee force-pushed the elasticsearch-9-support branch from dbd4a3a to 210c1f9 Compare March 17, 2026 20:52

mattmenefee marked this pull request as ready for review March 17, 2026 20:53

mattmenefee requested a review from a team as a code owner March 17, 2026 20:53

mattmenefee force-pushed the elasticsearch-9-support branch 2 times, most recently from c09dd71 to 98955eb Compare March 17, 2026 23:23

bbatsov reviewed Mar 18, 2026

View reviewed changes

mattmenefee force-pushed the elasticsearch-9-support branch 6 times, most recently from a4ee365 to 1fd450e Compare March 25, 2026 14:37

mattmenefee force-pushed the elasticsearch-9-support branch from 1fd450e to 3e2a303 Compare March 25, 2026 20:02

mattmenefee force-pushed the elasticsearch-9-support branch from 3e2a303 to e926d1a Compare April 2, 2026 19:53

mattmenefee force-pushed the elasticsearch-9-support branch 5 times, most recently from 5f4a7be to f6d8602 Compare June 1, 2026 22:03

mattmenefee mentioned this pull request Jun 1, 2026

Remove obsolete _type field references #1028

Merged

3 tasks

mattmenefee force-pushed the elasticsearch-9-support branch from f6d8602 to 82074ba Compare June 1, 2026 22:57

mattmenefee commented Jun 1, 2026

View reviewed changes

mattmenefee commented Jun 2, 2026

View reviewed changes

Comment thread .github/workflows/ruby.yml Outdated

AlfonsoUceda added the v9 label Jun 2, 2026

mattmenefee force-pushed the elasticsearch-9-support branch 2 times, most recently from 2dbb94e to 7a4ff81 Compare June 2, 2026 18:25

mattmenefee force-pushed the elasticsearch-9-support branch from 796ff44 to 7fd180a Compare June 4, 2026 16:51

mattmenefee force-pushed the elasticsearch-9-support branch from 7fd180a to aaf3615 Compare June 17, 2026 18:41

mattmenefee force-pushed the elasticsearch-9-support branch from aaf3615 to d19eb78 Compare June 30, 2026 22:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Elasticsearch 9.x support#1009

Add Elasticsearch 9.x support#1009
mattmenefee wants to merge 1 commit into
toptal:masterfrom
mattmenefee:elasticsearch-9-support

mattmenefee commented Mar 17, 2026 •

edited

Loading

Uh oh!

mattmenefee commented Mar 17, 2026 •

edited

Loading

Uh oh!

bbatsov left a comment

Uh oh!

AlfonsoUceda commented Mar 25, 2026

Uh oh!

mattmenefee commented Mar 25, 2026

Uh oh!

mattmenefee Jun 1, 2026

Uh oh!

Uh oh!

mattmenefee commented Jun 3, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

mattmenefee commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking changes

Test plan

Uh oh!

mattmenefee commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bbatsov left a comment

Choose a reason for hiding this comment

Uh oh!

AlfonsoUceda commented Mar 25, 2026

Uh oh!

mattmenefee commented Mar 25, 2026

Uh oh!

mattmenefee Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mattmenefee commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mattmenefee commented Mar 17, 2026 •

edited

Loading

mattmenefee commented Mar 17, 2026 •

edited

Loading

mattmenefee commented Jun 3, 2026 •

edited

Loading