Skip to content

perf(state_manager): [DSM-145] Flush large writes from background syncfs thread#10777

Open
alin-at-dfinity wants to merge 1 commit into
masterfrom
alin/DSM-145-background-syncfs-thread
Open

perf(state_manager): [DSM-145] Flush large writes from background syncfs thread#10777
alin-at-dfinity wants to merge 1 commit into
masterfrom
alin/DSM-145-background-syncfs-thread

Conversation

@alin-at-dfinity

@alin-at-dfinity alin-at-dfinity commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

The blocking syncfs in finalize_and_remove_unverified_marker flushes the dirty data produced earlier in the checkpoint (notably the freshly serialized per-canister protos). At high canister counts this is ~15s on the CUP-critical path. Previously it was cheap only because the overlay/Wasm validation (being removed in #10776) happened to give background writeback time to drain that data first.

Spawn a dedicated TipSyncfs thread that the tip thread pings (non-blocking, bounded(1) + try_send, self-coalescing) after every large batch of file writes/hardlinks: after serializing Wasm binaries and protos, after flushing a round's page-map deltas, and after reset_tip's hardlink tree and the merge. The kernel then flushes that data concurrently with the rest of the checkpoint, so the blocking syncfs in finalize finds little left to do (~15s -> ~0.1s at 200k canisters).

finalize's syncfs stays as the durability barrier, so the background syncfs calls are purely an optimization and their errors are non-fatal.

…cfs thread

The blocking `syncfs` in `finalize_and_remove_unverified_marker` flushes the dirty data produced earlier in the checkpoint (notably the freshly serialized per-canister protos). At high canister counts this is ~15s on the CUP-critical path. Previously it was cheap only because the overlay/Wasm validation (being removed in #10776) happened to give background writeback time to drain that data first.

Spawn a dedicated `TipSyncfs` thread that the tip thread pings (non-blocking, `bounded(1)` + `try_send`, self-coalescing) after every large batch of file writes/hardlinks: after serializing Wasm binaries and protos, after flushing a round's page-map deltas, and after `reset_tip`'s hardlink tree and the merge. The kernel then flushes that data concurrently with the rest of the checkpoint, so the blocking `syncfs` in `finalize` finds little left to do (~15s -> ~0.1s at 200k canisters).

`finalize`'s `syncfs` stays as the durability barrier, so the pings are purely an optimization and their errors are non-fatal.
@alin-at-dfinity
alin-at-dfinity requested a review from a team as a code owner July 14, 2026 16:47
@github-actions github-actions Bot added the perf label Jul 14, 2026
@zeropath-ai

zeropath-ai Bot commented Jul 14, 2026

Copy link
Copy Markdown

No security or compliance issues detected. Reviewed everything up to 0961b3b.

Security Overview
Detected Code Changes
Change Type Relevant files
Enhancement ► rs/state_layout/src/state_layout.rs
    Add syncfs method on StateLayout and internal syncfs helper supporting Linux and fallback to sync_all on non-Linux platforms
► rs/state_manager/src/tip.rs
    Spawn background TipSyncfs thread to perform non-fatal background syncfs calls after writes
    Trigger background syncfs on various tip operations to flush filesystem changes earlier

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant