feat: add read-optimized system table#403
Conversation
gripleaf
force-pushed
the
feat/value_stats_filter
branch
3 times, most recently
from
7aa2715 to
1449d94
Compare
gripleaf
changed the title
gripleaf
force-pushed
the
feat/value_stats_filter
branch
from
1449d94 to
b125cf8
Compare
|
@suxiaogang223 Would you mind helping take a look when you have time, since you’re most familiar with system table. Thanks a lot! |
| Status WriteAndFullCompact(TestHelper* helper, std::unique_ptr<RecordBatch>&& batch, | ||
| int64_t commit_identifier) { | ||
| PAIMON_RETURN_NOT_OK(helper->write_->Write(std::move(batch))); | ||
| PAIMON_RETURN_NOT_OK( |
There was a problem hiding this comment.
Could you place the output parameter at the end of the parameter list?
| .ok()); | ||
| ASSERT_TRUE(result.array->Equals(expected)) << result.array->ToString(); | ||
| } | ||
|
|
There was a problem hiding this comment.
Could you add a test case to verify that system table reads support partial-column projection and that predicate pushdown is effective? For the predicate setup, you can refer to NestedColumnPruningInteTest.NestedStructMapSelectedKeysWithPredicate. If the Parquet/ORC row group or stripe boundaries are configured properly, the test should be able to observe and validate that predicate pushdown is actually taking effect.
| /// @note If both SetReadFieldIds() and SetReadFieldNames() are set, both values are | ||
| /// preserved in ReadContext. InternalReadContext decides precedence. | ||
| ReadContextBuilder& SetReadFieldIds(const std::vector<int32_t>& read_field_ids); | ||
|
|
There was a problem hiding this comment.
Please keep this as-is — it is meant to notify users about the internal behavior.
| TestHelper::Create(dir->Str(), schema, | ||
| /*partition_keys=*/{}, | ||
| /*primary_keys=*/{}, options, | ||
| /*is_streaming_mode=*/true)); |
There was a problem hiding this comment.
What I’m really trying to test here is predicate behavior on PK tables, because RO tables additionally enable EnableValueFilter, which could affect the result.
| /*expected_commit_messages=*/std::nullopt)); | ||
|
|
||
| ASSERT_OK_AND_ASSIGN(SystemTableReadResult result, | ||
| ReadSystemTable(table_path + "$ro", options, /*streaming_mode=*/true)); |
There was a problem hiding this comment.
please call ReadSystemTable multiple times if you want to test StreamingScan
3 participants
Purpose
Add support for the read-optimized
$rosystem table. For primary-key tables,$roreads only the highest LSM level produced by full compaction and skips merge-on-read across levels. Append-only tables keep the same read behavior as the base table, including streaming reads.This also refactors read-optimized scan configuration into
DataTableBatchScan, uses context builders for$roscan/read contexts, and documents freshness and value-stats filtering behavior.Tests
E2E test:
SystemTableTest.TestReadOptimizedSystemTableRegistrationSystemTableTest.TestReadOptimizedSystemTablePathParsingTableScanTest.TestReadOptimizedPrimaryKeyStreamingScanUnsupportedSystemTableReadInteTest.TestReadOptimizedSystemTableSystemTableReadInteTest.TestReadOptimizedAppendOnlySystemTableWithStreamingScanSystemTableReadInteTest.TestReadOptimizedSystemTableWithBranchSystemTableReadInteTest.TestReadOptimizedSystemTableWithFirstRowMergeEngineAPI and Format
No public API, storage format, or protocol changes.
Documentation
Added user guide documentation for system tables and the read-optimized $ro system table.
Generative AI tooling
Generated-by: OpenAI Codex GPT-5