fix: HashTable dynamic rehash and group count limit for GROUP BY aggregation by MrPresent-Han · Pull Request #48174 · milvus-io/milvus

MrPresent-Han · 2026-03-10T09:09:40Z

Summary

HashTable now dynamically rehashes (doubles capacity) when load factor exceeds 7/8, fixing crash when GROUP BY cardinality > ~1792
Added configurable queryNode.segcore.maxGroupByGroups (default 100K) to cap total groups and prevent OOM on both C++ (per-segment HashTable) and Go (cross-segment agg reducer) layers
Added 4 C++ unit tests covering rehash basic/correctness, max groups limit, and multiple rehash rounds

issue: #47569

Test plan

C++ unit tests: --gtest_filter="*HashTableRehash*:*MaxGroups*"
E2E: GROUP BY aggregation with >2K unique values should succeed
E2E: Set queryNode.segcore.maxGroupByGroups to small value, verify clear error message

🤖 Generated with Claude Code

sre-ci-robot · 2026-03-10T09:10:44Z

[ci-v2-notice]
Notice: New ci-v2 system is enabled for this PR.

To rerun ci-v2 checks, comment with:

/ci-rerun-code-check // for ci-v2/code-check
/ci-rerun-build // for ci-v2/build
/ci-rerun-build-all // for ci-v2/build-all (multi-arch builds)
/ci-rerun-ut-integration // for ci-v2/ut-integration, will rerun ci-v2/build
/ci-rerun-ut-go // for ci-v2/ut-go, will rerun ci-v2/build
/ci-rerun-ut-cpp // for ci-v2/ut-cpp
/ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp, will rerun ci-v2/build
/ci-rerun-e2e-arm // for ci-v2/e2e-arm
/ci-rerun-e2e-default // for ci-v2/e2e-default
/ci-rerun-ciloop // for ci-v2/ciloop (build + unit tests in one pipeline)

If you have any questions or requests, please contact @zhikunyao.

mergify · 2026-03-10T10:52:15Z

@MrPresent-Han go-sdk check failed, comment rerun go-sdk can trigger the job again.

codecov · 2026-03-10T11:21:56Z

Codecov Report

❌ Patch coverage is 92.20779% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.60%. Comparing base (aed7c8b) to head (d694f37).
⚠️ Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/core/src/segcore/segcore_init_c.cpp	0.00%	4 Missing ⚠️
internal/agg/aggregate_reducer.go	75.00%	1 Missing and 2 partials ⚠️
pkg/util/paramtable/component_param.go	76.92%	2 Missing and 1 partial ⚠️
internal/core/src/exec/HashTable.cpp	96.42%	1 Missing ⚠️
internal/core/src/segcore/SegcoreConfig.h	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #48174      +/-   ##
==========================================
- Coverage   77.60%   77.60%   -0.01%     
==========================================
  Files        2107     2107              
  Lines      349666   349809     +143     
==========================================
+ Hits       271368   271465      +97     
- Misses      70035    70068      +33     
- Partials     8263     8276      +13

Components	Coverage Δ
Client	`79.25% <ø> (ø)`
Core	`83.98% <95.23%> (+0.02%)`	⬆️
Go	`75.77% <78.57%> (-0.02%)`	⬇️

Files with missing lines	Coverage Δ
internal/core/src/exec/HashTable.h	`98.14% <100.00%> (+7.58%)`	⬆️
...l/core/src/exec/operator/query-agg/GroupingSet.cpp	`100.00% <100.00%> (ø)`
internal/core/unittest/test_query_group_by.cpp	`99.88% <100.00%> (+0.01%)`	⬆️
internal/util/initcore/query_node.go	`79.23% <100.00%> (+0.49%)`	⬆️
internal/core/src/exec/HashTable.cpp	`97.35% <96.42%> (+3.85%)`	⬆️
internal/core/src/segcore/SegcoreConfig.h	`80.95% <50.00%> (-1.55%)`	⬇️
internal/agg/aggregate_reducer.go	`59.00% <75.00%> (+0.69%)`	⬆️
pkg/util/paramtable/component_param.go	`97.21% <76.92%> (-0.05%)`	⬇️
internal/core/src/segcore/segcore_init_c.cpp	`17.59% <0.00%> (-0.68%)`	⬇️

... and 40 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

MrPresent-Han · 2026-03-12T12:29:25Z

has verified by cc reviewers team

mergify · 2026-03-16T13:06:56Z

@MrPresent-Han go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2026-03-20T02:54:36Z

@MrPresent-Han go-sdk check failed, comment rerun go-sdk can trigger the job again.

xiaofan-luan · 2026-03-20T09:09:18Z

configs/milvus.yaml

      buildParallelRate: 0.5 # the ratio of building interim index parallel matched with cpu num
    multipleChunkedEnable: true # Deprecated. Enable multiple chunked search
    enableGeometryCache: false # Enable geometry cache for geometry data
+    maxGroupByGroups: 100000 # Maximum number of groups allowed in GROUP BY aggregation, enforced both per segment and during cross-segment merge. Exceeding this limit fails the query.


proxy might use this config? should put it somewhere or seprate to two configs

xiaofan-luan · 2026-03-20T09:34:16Z

/lgtm
/approve

sre-ci-robot · 2026-03-20T09:34:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liliu-z, MrPresent-Han, xiaofan-luan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [liliu-z,xiaofan-luan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

MrPresent-Han · 2026-03-22T08:07:13Z

/ci-rerun-ciloop

…egation (milvus-io#47569) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: MrPresent-Han <chun.han@gmail.com>

mergify · 2026-03-23T04:25:29Z

@MrPresent-Han go-sdk check failed, comment rerun go-sdk can trigger the job again.

MrPresent-Han · 2026-03-23T06:22:49Z

rerun go-sdk

liliu-z

/lgtm

…egation (milvus-io#48174) ## Summary - HashTable now dynamically rehashes (doubles capacity) when load factor exceeds 7/8, fixing crash when GROUP BY cardinality > ~1792 - Added configurable `queryNode.segcore.maxGroupByGroups` (default 100K) to cap total groups and prevent OOM on both C++ (per-segment HashTable) and Go (cross-segment agg reducer) layers - Added 4 C++ unit tests covering rehash basic/correctness, max groups limit, and multiple rehash rounds issue: milvus-io#47569 ## Test plan - [ ] C++ unit tests: `--gtest_filter="*HashTableRehash*:*MaxGroups*"` - [ ] E2E: GROUP BY aggregation with >2K unique values should succeed - [ ] E2E: Set `queryNode.segcore.maxGroupByGroups` to small value, verify clear error message 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: heihutu <heihutu@gmail.com>

…egation (milvus-io#48174) ## Summary - HashTable now dynamically rehashes (doubles capacity) when load factor exceeds 7/8, fixing crash when GROUP BY cardinality > ~1792 - Added configurable `queryNode.segcore.maxGroupByGroups` (default 100K) to cap total groups and prevent OOM on both C++ (per-segment HashTable) and Go (cross-segment agg reducer) layers - Added 4 C++ unit tests covering rehash basic/correctness, max groups limit, and multiple rehash rounds issue: milvus-io#47569 ## Test plan - [ ] C++ unit tests: `--gtest_filter="*HashTableRehash*:*MaxGroups*"` - [ ] E2E: GROUP BY aggregation with >2K unique values should succeed - [ ] E2E: Set `queryNode.segcore.maxGroupByGroups` to small value, verify clear error message 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Chun Han <116052805+MrPresent-Han@users.noreply.github.com>

sre-ci-robot requested review from czs007 and godchen0212 March 10, 2026 09:09

sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Mar 10, 2026

mergify bot added dco-passed DCO check passed. kind/bug Issues or changes related a bug labels Mar 10, 2026

sre-ci-robot added the low-code-coverage add test-label from zhikun, diff coverage > 80% label Mar 10, 2026

MrPresent-Han force-pushed the hashtable-rehash-master branch 2 times, most recently from 4bb6ab8 to c1e903b Compare March 11, 2026 04:05

sre-ci-robot removed the low-code-coverage add test-label from zhikun, diff coverage > 80% label Mar 11, 2026

mergify bot added the ci-passed label Mar 11, 2026

MrPresent-Han force-pushed the hashtable-rehash-master branch from c1e903b to d47f238 Compare March 12, 2026 02:21

sre-ci-robot added low-code-coverage add test-label from zhikun, diff coverage > 80% and removed ci-passed low-code-coverage add test-label from zhikun, diff coverage > 80% labels Mar 12, 2026

mergify bot added the ci-passed label Mar 12, 2026

MrPresent-Han force-pushed the hashtable-rehash-master branch from d47f238 to 8330542 Compare March 12, 2026 12:28

sre-ci-robot added low-code-coverage add test-label from zhikun, diff coverage > 80% and removed ci-passed low-code-coverage add test-label from zhikun, diff coverage > 80% labels Mar 12, 2026

mergify bot added the ci-passed label Mar 12, 2026

MrPresent-Han force-pushed the hashtable-rehash-master branch from 8330542 to d75ad20 Compare March 16, 2026 12:51

sre-ci-robot removed the ci-passed label Mar 16, 2026

MrPresent-Han force-pushed the hashtable-rehash-master branch from d75ad20 to 010ea9a Compare March 17, 2026 02:21

sre-ci-robot added low-code-coverage add test-label from zhikun, diff coverage > 80% and removed low-code-coverage add test-label from zhikun, diff coverage > 80% labels Mar 17, 2026

MrPresent-Han force-pushed the hashtable-rehash-master branch from ed7eb41 to 52954ae Compare March 20, 2026 03:08

sre-ci-robot added low-code-coverage add test-label from zhikun, diff coverage > 80% and removed lgtm labels Mar 20, 2026

xiaofan-luan reviewed Mar 20, 2026

View reviewed changes

sre-ci-robot assigned xiaofan-luan Mar 20, 2026

sre-ci-robot added the lgtm label Mar 20, 2026

MrPresent-Han force-pushed the hashtable-rehash-master branch from 52954ae to 0a4fc85 Compare March 22, 2026 02:59

sre-ci-robot removed lgtm low-code-coverage add test-label from zhikun, diff coverage > 80% labels Mar 22, 2026

MrPresent-Han force-pushed the hashtable-rehash-master branch 3 times, most recently from 002a0c4 to b156e18 Compare March 22, 2026 09:00

sre-ci-robot added the low-code-coverage add test-label from zhikun, diff coverage > 80% label Mar 22, 2026

MrPresent-Han force-pushed the hashtable-rehash-master branch from b156e18 to a3b47e8 Compare March 22, 2026 11:46

fix: HashTable dynamic rehash and group count limit for GROUP BY aggr…

d694f37

…egation (milvus-io#47569) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: MrPresent-Han <chun.han@gmail.com>

MrPresent-Han force-pushed the hashtable-rehash-master branch from a3b47e8 to d694f37 Compare March 23, 2026 02:18

sre-ci-robot removed the low-code-coverage add test-label from zhikun, diff coverage > 80% label Mar 23, 2026

mergify bot added the ci-passed label Mar 23, 2026

liliu-z reviewed Mar 23, 2026

View reviewed changes

sre-ci-robot added the lgtm label Mar 23, 2026

sre-ci-robot merged commit cc9c60e into milvus-io:master Mar 23, 2026
18 of 23 checks passed

NicoYuan1986 mentioned this pull request Mar 24, 2026

[Bug]: [Nightly] Woodpecker flush regression after WP v0.1.25 upgrade — 82 flush failures in master #648 #48481

Closed

Conversation

MrPresent-Han commented Mar 10, 2026

Summary

Test plan

Uh oh!

sre-ci-robot commented Mar 10, 2026

Uh oh!

mergify bot commented Mar 10, 2026

Uh oh!

codecov bot commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

MrPresent-Han commented Mar 12, 2026

Uh oh!

mergify bot commented Mar 16, 2026

Uh oh!

mergify bot commented Mar 20, 2026

Uh oh!

xiaofan-luan Mar 20, 2026

Choose a reason for hiding this comment

Uh oh!

xiaofan-luan commented Mar 20, 2026

Uh oh!

sre-ci-robot commented Mar 20, 2026

Uh oh!

MrPresent-Han commented Mar 22, 2026

Uh oh!

mergify bot commented Mar 23, 2026

Uh oh!

MrPresent-Han commented Mar 23, 2026

Uh oh!

liliu-z left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Mar 10, 2026 •

edited

Loading