branch-4.1: [feature](cloud) Support file cache write index only#65268
Open
bobhan1 wants to merge 1 commit into
Open
branch-4.1: [feature](cloud) Support file cache write index only#65268bobhan1 wants to merge 1 commit into
bobhan1 wants to merge 1 commit into
Conversation
This PR adds a new BE config, `enable_file_cache_write_index_file_only`, to make cloud rowset writes and compaction outputs write only index-related content into file cache. This avoids actively putting segment data into file cache during write, reducing cache pollution from large data pages. The main target scenario is a limited local cache where query performance should prioritize indexes and segment metadata. With this mode enabled, the write path protects inverted index files and segment index/footer ranges, while normal data pages are still loaded and cached by the query read path when needed. - Add BE config `enable_file_cache_write_index_file_only`, default `false`. - Make rowset/segment file-cache write decisions file-type aware: - `SEGMENT_FILE` does not create a full-file cache builder and does not allow adaptive write in index-only mode. - `INVERTED_INDEX_FILE` is still written into file cache through the direct write path. - segment index/footer ranges are preloaded into file cache by `SegmentIndexFileCacheLoader` after segment close. - Update packed file handling to honor the original file type when deciding whether to write file cache. - Add profile statistics for segment footer/index reads and separate them from data-page and inverted-index reads. - Replace correlated boolean parameters in `CachedRemoteFileReader::_update_stats` with a read-type enum: - data page - inverted index - segment footer/index - Add BE UTs and cloud regressions for both normal file and packed file paths. When `enable_file_cache_write_index_file_only=true`: - Segment data is not actively written into file cache during write, flush, or compaction output. - Segment index/footer ranges are synchronously preloaded into file cache after segment close. - Independent inverted index files are written into file cache through the direct write path. - Query-side data-page reads keep the existing behavior. The query read path may still fill cache according to current rules. - Segment open, footer/index access, and inverted index access can hit the index-related content preloaded by the write path. - Query profile exposes separate cache/remote read statistics for data pages, inverted indexes, and segment footer/index reads. The write-side file-cache behavior is decided in this order: 1. `enable_file_cache=false` is the highest-priority global switch. - All file-cache writes are disabled. - This includes independent inverted index direct cache and segment index/footer preload. 2. `enable_file_cache=true` and `enable_file_cache_write_index_file_only=true` enables global forced index-only mode. - Segment data does not create a full-file cache builder. - Segment data cannot be written by adaptive write. - Segment index/footer ranges are written into file cache by `SegmentIndexFileCacheLoader` after segment close. - Independent inverted index files are written through the `INVERTED_INDEX_FILE` direct write path. - This config has higher priority than `enable_file_cache_write_base_compaction_index_only` and `enable_file_cache_write_cumu_compaction_index_only`. - These index writes do not depend on `enable_file_cache_adaptive_write`, compaction keep configs, hit-ratio thresholds, or request/session `write_file_cache`. 3. `enable_file_cache=true`, `enable_file_cache_write_index_file_only=false`, and a compaction-level index-only config is enabled applies index-only behavior only to the matching compaction output. - `enable_file_cache_write_base_compaction_index_only=true` only forces base compaction output to use index-only file-cache writes. - `enable_file_cache_write_cumu_compaction_index_only=true` only forces cumulative compaction output to use index-only file-cache writes. - These two configs do not affect load, normal flush, schema change output, or query read-path cache fill. - Compaction types not matched by these configs continue to use existing `write_file_cache`, adaptive write, compaction keep, and hit-ratio decisions. 4. `enable_file_cache=true` and all three index-only configs are `false` preserves existing behavior. - `write_file_cache`, `enable_file_cache_adaptive_write`, compaction keep configs, schema-change hit ratios, and existing rowset/file-writer decisions continue to control whether cache is written. - When index-only is disabled, `enable_file_cache_adaptive_write=true` keeps the existing adaptive write behavior. Segment data may be written into file cache if the cache-space check passes. - When index-only is enabled, adaptive write no longer participates in segment data decisions. - Even if `enable_file_cache_adaptive_write=true`, segment data is still not written into file cache, while segment index/footer and independent inverted index files are still written. Related configs include: - `enable_file_cache_keep_base_compaction_output` - `file_cache_keep_base_compaction_output_min_hit_ratio` - compaction output hit-ratio checks When index-only is disabled, existing compaction output cache behavior is preserved. When index-only is enabled, compaction output is forced into index-only behavior: - Compaction output segment data is not actively written into file cache. - Segment index/footer ranges are synchronously written into file cache. - Independent inverted index files are direct-cached. - Compaction keep configs and hit-ratio thresholds no longer decide whether index content is written into file cache. Related configs include: - `enable_file_cache_write_base_compaction_index_only` - `enable_file_cache_write_cumu_compaction_index_only` These two configs are narrower compaction-output policies and only take effect when `enable_file_cache_write_index_file_only=false`. - When `enable_file_cache_write_base_compaction_index_only=true`, base compaction output uses index-only behavior: output segment data is not actively cached, while output segment index/footer ranges and independent inverted index files are written into file cache. - When `enable_file_cache_write_cumu_compaction_index_only=true`, cumulative compaction output uses index-only behavior: output segment data is not actively cached, while output segment index/footer ranges and independent inverted index files are written into file cache. - If only one of the two configs is enabled, the other compaction type keeps the existing cache-write decision flow. - If `enable_file_cache_write_index_file_only=true`, the global index-only config takes precedence and the base/cumulative compaction-specific configs no longer change the final behavior. - `enable_file_cache=false` is still the highest-priority switch and disables all of these file-cache writes. - When index-only is disabled, `file_cache_keep_schema_change_output_min_hit_ratio` keeps its existing semantics. - When index-only is enabled, schema change output also follows index-only behavior: - segment data is not actively written into file cache; - index-related content is written into file cache; - schema-change hit-ratio thresholds no longer decide whether index content is cached. - `enable_file_cache=false` disables everything and has the highest priority. - When index-only is enabled, the BE config has higher priority than request/session `write_file_cache=false`. - Even if session `disable_file_cache=true` makes request-side `write_file_cache=false`, as long as BE `enable_file_cache=true` and index-only is enabled: - index-related content is still written into file cache; - segment data is still not written into file cache. - When index-only is disabled, `disable_file_cache=true` preserves existing behavior and load output does not actively write file cache. - When index-only is disabled, packed small files keep the existing whole-file cache behavior. - When index-only is enabled: - packed small segment files must not be whole-file cached, preventing segment data from entering file cache; - segment index/footer ranges are written into file cache by `SegmentIndexFileCacheLoader`; - packed independent index small files are independent index files and can be whole-file cached. - Segment index/footer ranges are written through synchronous dry-run reads and do not depend on write-path `enable_flush_file_cache_async`. - Independent inverted index files still use the direct write path, so they continue to follow existing async flush behavior. `SegmentIndexFileCacheLoader` marks index-related reads as index data so that they go to the `FileCacheType::INDEX` queue when possible. After enabling index-only mode, `index_percent` should be reviewed based on index size. The default `index_percent=5` may be insufficient for large indexes. | Scenario | index-only=false | index-only=true | | --- | --- | --- | | load segment data | follows existing cache policy | not actively cached | | load segment index/footer | depends on existing whole-file cache or read path | synchronously range-read into file cache after segment close | | load independent inverted index file | follows existing cache policy | direct-cached by write path | | cumulative compaction segment data | may be cached by existing behavior | not actively cached | | cumulative compaction segment index/footer | may be cached by existing behavior | synchronously range-read into file cache after segment close | | base compaction force-keep output | whole output rowset may be cached | only output segment index/footer is preloaded | | adaptive write has enough space | segment data may be cached | segment data is still not cached | | packed small index file | may be cached | can be whole-file cached | | packed small segment file | may be cached | not whole-file cached; internal index/footer is preloaded | | query read fill cache | unchanged | unchanged | | event-driven warm-up | unchanged | unchanged | Add BE config `enable_file_cache_write_index_file_only` to support index-only file-cache writes for cloud rowset writes and compaction outputs. (cherry picked from commit 73c481f)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
bobhan1
changed the title
Contributor
Author
|
run buildall |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: #64995
Problem Summary:
Pick master commit
73c481f65848ed5f449fad2ac9e7c59cf54bcf9etobranch-4.1to support file cache write index only in cloud mode.Adaptations for
branch-4.1:IndexFileWriterconstructor shape onbranch-4.1.row_binlog_segment_writer.hinclude.Validation note:
branch-4.1does not have that docker-compose fix yet.Release note
Support file cache write index only in cloud mode.
Check List (For Author)
Manual test:
git diff --check upstream/branch-4.1...HEAD./run-be-ut.sh --run --filter=CloudFileCacheWriteIndexOnly* -j100./build.sh --be --fe --cloud -j100docker build -f docker/runtime/doris-compose/Dockerfile -t bh-cluster-2 ../run-regression-test.sh --run -d regression-test/suites/cloud_p0/cache/write_index_only -g docker -runMode=cloud -dockerSuiteParallel 1Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)