Skip to content

feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns)#375

Merged
lxy-9602 merged 62 commits into
alibaba:mainfrom
zhf999:paged-bitmap2
Jul 10, 2026
Merged

feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns)#375
lxy-9602 merged 62 commits into
alibaba:mainfrom
zhf999:paged-bitmap2

Conversation

@zhf999

@zhf999 zhf999 commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Purpose

This PR follows PR #371.

  • Support pushing selection bitmaps down to Parquet page-level filtering so only matching pages are read when both predicate and bitmap pruning are available.
  • Refactor target row-group filtering metadata into a dedicated TargetRowGroup type to carry row-group index, partial-match status, row ranges, and read-range exclusion flags consistently across planning and reading phases.

Changes

  • src/paimon/format/parquet/target_row_group.h

    • Introduced TargetRowGroup class and TargetRowGroups alias.
    • Replaced ad-hoc struct usage from row_ranges.h.
  • src/paimon/format/parquet/parquet_file_batch_reader.h

    • Updated row-group filtering APIs to operate on TargetRowGroups instead of raw std::vector<int32_t>.
  • src/paimon/format/parquet/parquet_file_batch_reader.cpp

    • In SetReadSchema, switched row-group candidate list to TargetRowGroups and applied bitmap filter before page-index filtering.
    • Added bitmap-driven row-group/page pruning:
      • If any nested columns are included in read schema, bitmap filtering fallbacks to row-group-level.
      • Translate bitmap to row_ranges to filter page with bitmap.
      • The translation strategy follows Paimon, which prunes unneeded rows at the start and end of each page.

Tests

  • Added bitmap + page-level integration tests:
    • BitmapAllPagesSomeRowGroups
    • BitmapPartialPagesSingleRowGroup
    • BitmapAllAndPartialPagesMixed
    • BitmapAllPagesWithPredicate
    • BitmapPartialPagesWithPredicate
    • BitmapMixedWithPredicate
    • NestedMapBitmapFallback
    • NestedListBitmapFallback
    • NestedStructBitmapFallback
  • Coverage includes bitmap-only pruning, bitmap+predicate pruning, and mixed partial/full page hit cases.

Datasets

Multiple performance testing were conducted on 2 dataset.

Dataset File size num_rows num_columns num_rowgroups page length (in rows)
Dataset#1 6.7GB 2M 1,176 39 1024
Dataset#2 3GB 11M 4 23 1024

Test Methodology

To validate bitmap behavior, we organize tests into six groups based on bitmap distribution patterns. Each group contains five distinct datasets, and the final result reported is the average of these five runs.

Test Cases

  1. Matching records are contiguous and only one row is queried.
  2. Matching records are contiguous and 100 rows are queried.
  3. Matching records are contiguous and 10,000 rows are queried.
  4. Matching records are scattered in 5 segments, each segment containing 1 row.
  5. Matching records are scattered in 5 segments, each segment containing 100 rows.
  6. Matching records are scattered in 100 segments, each segment containing 1 row.
  7. Scattered bitmap, skip 1 row, read 1 row, skip 1 row, read 1 row until EOF (i.e. 1010101010...).
  8. Scattered bitmap, skip 99 rows , read 1 row, skip 99 rows, skip 1 row until EOF.
  9. Scattered bitmap, skip 9999 rows , read 1 row, skip 9999 rows, skip 1 row until EOF.

Test Results

On Dataset#1

Branch Case 1 Case 2 Case3 Case 4 Case 5 Case 6 Case 7 Case 8 Case 9
Main 2.27s 2.28s 2.19s 8.74ss 8.75s 105.58s 126.08s 125.96s 126.00s
This PR 0.33s 0.32s 0.90s 1.38s 1.35s 15.82s 126.81s 116.24s 20.77s

On Dataset#2

Branch Case 1 Case 2 Case3 Case 4 Case 5 Case 6 Case 7 Case 8 Case 9
Main 1.30s 1.26s 1.26s 2.77s 2.76s 30.48s 31.27s 31.28s 31.28s
This PR 8ms 7ms 39ms 25ms 26ms 0.46s 35.18s 32.31s 1.99s

API and Format

  • No public API changes.
  • No storage format/protocol changes.

Documentation

  • No user-facing docs required for this change.

Generative AI tooling

  • gpt-5.3-codex

Copilot AI review requested due to automatic review settings June 17, 2026 07:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@zhf999 zhf999 changed the title feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) [WIP] feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) Jun 18, 2026
@zhf999 zhf999 changed the title [WIP] feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) Jun 18, 2026
Comment thread src/paimon/format/parquet/parquet_file_batch_reader.cpp Outdated
Comment thread src/paimon/format/parquet/parquet_file_batch_reader.cpp
@zhf999 zhf999 changed the title feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) [WIP] feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) Jun 24, 2026
@zhf999 zhf999 changed the title [WIP] feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) feat(parquet): support pushing bitmaps down to page-level filtering (except for nested columns) Jun 29, 2026
@zhf999 zhf999 marked this pull request as ready for review June 29, 2026 08:30
@zhf999 zhf999 marked this pull request as draft June 29, 2026 10:18
@zhf999 zhf999 marked this pull request as ready for review July 6, 2026 09:34
Comment thread src/paimon/format/parquet/parquet_file_batch_reader.h Outdated
Comment thread src/paimon/common/reader/prefetch_file_batch_reader_impl.cpp
Comment thread src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp Outdated
Comment thread src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp
Comment thread src/paimon/format/parquet/parquet_file_batch_reader.cpp
Comment thread src/paimon/format/parquet/parquet_file_batch_reader.cpp

@zjw1111 zjw1111 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-up work on page-level bitmap pushdown, and for the solid test coverage. A few follow-up nits on stale comments and naming, on top of the points already raised in the earlier review.

Comment thread src/paimon/format/parquet/parquet_file_batch_reader.cpp Outdated
Comment thread src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp Outdated
Comment thread src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp Outdated
Comment thread src/paimon/format/parquet/target_row_group.h Outdated
Comment thread src/paimon/format/parquet/parquet_file_batch_reader.h Outdated
Comment thread src/paimon/format/parquet/target_row_group.h Outdated
@zhf999 zhf999 requested review from lxy-9602 and zjw1111 July 9, 2026 10:10

@zjw1111 zjw1111 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick updates — most of the earlier comments are addressed. Two small follow-ups below.

Comment thread src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp Outdated
Comment thread src/paimon/format/parquet/parquet_file_batch_reader.cpp Outdated

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@zjw1111 zjw1111 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough follow-ups and the added test coverage. All review comments are addressed. LGTM.

@lxy-9602 lxy-9602 merged commit 3483316 into alibaba:main Jul 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants