[fix](iceberg) Project Iceberg system table scans#65262
Open
suxiaogang223 wants to merge 1 commit into
Open
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
|
what will happen if |
Contributor
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review completed for PR 65262.
I did not find a substantiated correctness issue to raise as an inline comment.
Critical checkpoint conclusions:
- Goal and tests: the change projects required top-level columns only for Iceberg system table scans before SDK planning, while normal Iceberg data-table scans keep the existing BE scan-range pruning path. The added external regression covers
$data_filesand$filesprojected queries for ORC and Parquet tables containingmap<boolean, boolean>. - Scope: the code change is narrow and localized to
IcebergScanNode.createTableScan()plus the helper used to build the system-table projection list. - Concurrency/lifecycle: no new shared mutable lifecycle or lock-sensitive path was introduced. The scan still plans system-table splits through the existing synchronous system-table path and caches the planned scan as before.
- Compatibility/config/session: no new config, persisted format, FE-BE protocol field, or rolling-upgrade-sensitive thrift field was added. The existing JNI scanner required-field contract remains name based.
- Parallel paths: normal Iceberg table scans are intentionally not changed; system-table scans are the path where Iceberg SDK
DataTaskmaterializes metadata rows from the serializedFileScanTaskschema. - Test/output coverage: the test uses existing external Iceberg suite patterns with direct SQL result assertions for nondeterministic file sizes and orders result sets where needed. No
.outfile is required for these assertion-only checks. - Validation limits: this runner checkout is grafted/shallow with no parent commit,
.worktree_initializedis missing, andthirdparty/installed/bin/protocis unavailable, so I could not run the FE build or regression suite locally. I did run static PR-scope checks against the GitHub patch for changed-file scope and whitespace/CRLF issues.
User focus: no additional user-provided review focus was supplied.
Subagent conclusions:
optimizer-rewrite: no candidates in the initial pass; convergence round 1 returnedNO_NEW_VALUABLE_FINDINGS.tests-session-config: no candidates in the initial pass; convergence round 1 returnedNO_NEW_VALUABLE_FINDINGS.- No candidates were accepted, dismissed, or merged as duplicates, and the final inline comment set is empty.
Contributor
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Iceberg system table scans serialize SDK FileScanTask objects and the Java metadata scanner materializes rows from DataTask rows. Doris did not pass the actually required system table columns into the Iceberg SDK scan, so metadata tables such as files and data_files were planned with the full schema. For files metadata tables this can include virtual readable_metrics even when the query only requests columns such as file_size_in_bytes. The original select-based projection is also not enough because Iceberg DataTask implementations expose schema differently: manifest-backed tasks expose the projected schema, while StaticDataTask exposes the full schema even though rows are projected. Use an explicit SDK projected Schema whose prefix matches Doris required columns, append filter-only columns after that prefix, and make the JNI reader read projected rows by required ordinal while using the task schema only for field type lookup. Add regression coverage for ORC and Parquet projected files/data_files queries and for explicitly requested readable_metrics.
### Release note
None
### Check List (For Author)
- Test: Build
- DISABLE_BUILD_UI=ON ./build.sh --fe
- Behavior changed: No
- Does this need documentation: No
suxiaogang223
force-pushed
the
codex/iceberg-system-table-projection
branch
from
0dd8b91 to
b4a9dcb
Compare
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29640 ms |
Contributor
TPC-DS: Total hot run time: 179899 ms |
Contributor
ClickBench: Total hot run time: 24.93 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Iceberg system table scans serialize Iceberg SDK
FileScanTaskobjects and the Java metadata scanner materializes rows from the task schema. Doris did not pass the actually required system table columns into the Iceberg SDK scan, so metadata tables such as$filesand$data_fileswere planned with the full schema. For files metadata tables, the full schema can include virtualreadable_metricseven when the SQL only requests columns such asfile_size_in_bytes.This PR applies SDK top-level column projection only for Iceberg system table scans. Normal Iceberg data table scans continue to rely on Doris BE scan range params for column pruning.
What changed?
TableScan.select(...)projection inIcebergScanNodebefore planning the Iceberg SDK scan.map<boolean, boolean>columns, verifying projected$data_filesand$filesqueries.Check List (For Author)
DISABLE_BUILD_UI=ON MAVEN_OPTS='-Xmx8g' FE_MAVEN_THREADS=1 FE_MAVEN_RETRY_THREADS=1 ./build.sh --fe