[branch-4.1][feature](be) Support expression zonemap pruning#65263
[branch-4.1][feature](be) Support expression zonemap pruning#65263mrhhsg wants to merge 1 commit into
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
I found one correctness issue that can make expression ZoneMap pruning skip matching rows, so I am requesting changes.
Critical checkpoints:
- Existing review context: no prior inline review comments or replies were present, so M-1 is not a duplicate.
- Coverage: reviewed the changed expression ZoneMap evaluators, runtime-filter/null handling, internal segment/page pruning, Parquet row-group/page pruning, session/thrift propagation, and the added BE/regression tests.
- User focus: no additional user-provided review focus was supplied.
- Validation:
git diff --check 27eda17673797a507474b04263d26681b4cd1f0a f67267b7b0f3c6f009f8349647557d724ffb8318 --was clean. I did not run the full build or test suite in this runner. - Repo review skill: no repo-local code-review
SKILL.mdwas present in this checkout after hidden-path search.
Subagent conclusions:
- optimizer-rewrite initially found no candidates; after M-1 was added, convergence round 1 returned
NO_NEW_VALUABLE_FINDINGS. - tests-session-config initially found no candidates; after M-1 was added, convergence round 1 returned
NO_NEW_VALUABLE_FINDINGS. - Main review accepted M-1 as the only final inline comment.
| } | ||
|
|
||
| // Segment-level expr-zonemap runs before SegmentIterator can rebind storage expressions to | ||
| // the reader schema. Only apply it when scan tuple slot ordinals already match this schema. |
There was a problem hiding this comment.
This segment-level pruning is still using the original scan-tuple slot ordinals. build_segment_zonemap_context() maps collect_slot_column_ids() through schema.column_id(slot_index), but the expressions are only cloned/rebound later in SegmentIterator::_construct_compound_expr_context(). The new rebinding helper explicitly says AGG_KEYS and non-MOW UNIQUE_KEYS readers can expand the reader schema so the scan tuple ordinal is not the runtime reader-block ordinal. For those tables, a predicate on scan slot 0 can be evaluated here against schema.column_id(0) for a different column and skip an entire segment that actually has matching rows. Please either run this segment-level expr-zonemap pass on cloned/rebound expressions, or gate it to the key/table modes where the scan tuple layout is guaranteed to match the reader schema.
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
mrhhsg
force-pushed
the
backport-63389-expr-zonemap-branch-4.1
branch
from
f67267b to
0c4abde
Compare
Problem Summary: Support expression-based ZoneMap pruning for internal
and Parquet scan paths. Supported expressions include comparisons,
IN/NOT IN, IS NULL/IS NOT NULL, and starts_with. The implementation adds
segment/page/row-group pruning, conservative fallback semantics, and
profile counters, with BE unit and SQL regression coverage. CHAR string
range statistics are supported through the trimmed logical bounds
produced by ZoneMap deserialization, so CHAR predicates can participate
in expression ZoneMap pruning.
This follow-up also removes duplicated capability checks between
`can_evaluate_zonemap_filter` and `evaluate_zonemap_filter`. Capability
checks now focus on expression shape and NULL-literal cases, while
storage/runtime datatype invariants are asserted with `DORIS_CHECK` in
the evaluation path.
The latest follow-up centralizes HybridSet-to-Field materialization for
scalar and direct IN predicates, removes identity-cast unwrapping from
expression ZoneMap matching, and lets null-aware runtime filters use
ZoneMap pruning when the target zone has no NULL values while
conservatively keeping zones that may contain NULL.
The latest crash-fix follow-up makes comparison expression ZoneMap
capability checks conservative when the scan slot type and literal type
are not directly compatible, such as external table predicates comparing
`Nullable(DateTimeV2(6))` with a `DateTimeV2(0)` literal. In that case
Doris now skips expression ZoneMap pruning instead of triggering a BE
`DORIS_CHECK`, leaving the normal expression evaluation path to filter
rows.
Support expression-based ZoneMap pruning for scan predicates to skip
data using ZoneMap statistics.
- Test: Unit Test / Regression test / Format check
- DORIS_HOME=<repo> ninja -C be/ut_build_ASAN Exprs Storage Exec Format
doris_be_test
- DORIS_HOME=<repo> ninja -C be/ut_build_ASAN Exprs doris_be_test
- ./run-be-ut.sh --run --filter=SegmentFilterHelpersTest.*
- ./run-be-ut.sh --run
--filter=ScanNormalizePredicateTest.*:RuntimeFilterConsumerHelperTest.*:VRuntimeFilterWrapperSamplingTest.*:ScannerLateArrivalRFTest.*
- ./run-be-ut.sh --run --filter=ParquetExprTest.test_expr_zonemap_*
- ./run-be-ut.sh --run --filter=ExprZonemapFilterTest.*
- ./run-be-ut.sh --run
--filter='ExprZonemapFilterTest.FunctionStringStartsWithZonemapUsesPrefixRange:function_string_test.function_starts_with_test'
- ./run-be-ut.sh --run
--filter='ExprZonemapFilterTest.*:ParquetExprTest.test_expr_zonemap_*'
- ninja -C be/ut_build_ASAN
src/storage/CMakeFiles/Storage.dir/segment/segment.cpp.o
src/storage/CMakeFiles/Storage.dir/segment/segment_iterator.cpp.o
- ./run-be-ut.sh --run
--filter=ParquetExprTest.test_expr_zonemap_page_filter_keeps_unsupported_results_and_counts_stats
- doris-local-regression.sh all -d inverted_index_p0 -s
test_index_range_in_select
- ./run-regression-test.sh --conf <local-regression-conf> --run -d
query_p0/expr_zonemap -s test_expr_zonemap_pruning
- ./run-regression-test.sh --conf <local-regression-conf> --run -d
query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1
- doris-local-regression --network 10.26.20.3/24 all -d
query_p0/expr_zonemap -s test_expr_zonemap_pruning
- doris-local-regression --network 10.26.20.3/24 all -d
query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1
- doris-local-regression --network 10.26.20.3/24 all -d mtmv_p0 -s
test_iceberg_mtmv -conf enableIcebergTest=true -conf
iceberg_rest_uri_port=18283 -conf iceberg_minio_port=19001 -conf
externalEnvIp=127.0.0.1
- build-support/clang-format.sh --files
be/src/exprs/function/functions_comparison.h
be/test/exprs/expr_zonemap_filter_test.cpp
- build-support/check-format.sh --files
be/src/exprs/function/functions_comparison.h
be/test/exprs/expr_zonemap_filter_test.cpp
- build-support/clang-format.sh
- build-support/check-format.sh
- doris-local-regression --network 10.26.20.3/24 run -d
query_p0/expr_zonemap -s test_expr_zonemap_pruning
- doris-local-regression --network 10.26.20.3/24 run -d
query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1
- git diff --check
- Behavior changed: Yes. Supported scan predicates, including CHAR
string predicates, can now prune data with expression ZoneMap
evaluation.
- Does this need documentation: No
(cherry picked from commit 257417b)
mrhhsg
force-pushed
the
backport-63389-expr-zonemap-branch-4.1
branch
from
0c4abde to
7629d82
Compare
|
/review |
There was a problem hiding this comment.
Automated review completed for PR 65263.
I found no new non-duplicate inline comment to submit. The main correctness concern I saw is already covered by the existing inline thread on be/src/storage/segment/segment.cpp:350, so I did not re-submit the same issue.
Scope checked: expression zonemap evaluators, compound/IN/null/string comparison semantics, runtime-filter and direct-IN rewrites, OLAP segment/page pruning integration, Parquet row-group/page pruning integration, session/thrift propagation, profile counters, and the new BE/regression tests.
User focus: no additional user-provided review focus was supplied.
Subagent conclusions: optimizer-rewrite found no new evidence-backed candidates; tests-session-config found no new evidence-backed candidates. Convergence round 1 ended with both subagents replying NO_NEW_VALUABLE_FINDINGS for the same ledger and empty proposed final comment set.
Validation: static review against exact base/head SHAs and git diff --check passed. I did not run the full BE/FE test suites in this runner.
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
FE Regression Coverage ReportIncrement line coverage |
2 participants
What problem does this PR solve?
Issue Number: None
Related PR: #63389
Problem Summary:
Backport expression-based ZoneMap pruning to branch-4.1 for internal and Parquet scan paths. Supported expressions include comparisons, IN/NOT IN, IS NULL/IS NOT NULL, and starts_with. The implementation adds segment/page/row-group pruning, conservative fallback semantics, profile counters, BE unit coverage, and SQL regression coverage.
Release note
Support expression-based ZoneMap pruning for scan predicates.
Check List (For Author)
Validation:
./build.sh --be --fepassed.doris-local-regression run -d query_p0/expr_zonemap -s test_expr_zonemap_pruningpassed.doris-local-regression run -d query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1passed../run-be-ut.sh --run --filter='ExprZonemapFilterTest.*:SegmentIteratorExprZonemapTest.*:ParquetExprTest.test_expr_zonemap_*' -j 64passed../run-fe-ut.sh --run org.apache.doris.qe.SessionVariablesTest,org.apache.doris.qe.VariableMgrTestpassed.git diff HEAD^ HEAD --checkpassed.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)