Block Pruning prevent Sequencer to start and produce block #20130
Description
Describe the bug
When we trying to restart sequencer node, there is log about pruning MerkleChangeset, and it take ~20 minutes to complete, and in that 20min, the node just prune and it block the block producing flow
Here is the logs:
2025-12-03T13:38:04.046231835Z stdout F 2025-12-03T13:38:04.046176Z DEBUG pruner: Pruner finished tip_block_number=29679134 elapsed=1149.889397891s deleted_entries=16665479 limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 18446744073709551615, deleted: 16665479 }), time_limit: None } output=PrunerOutput { progress: Finished, segments: [(MerkleChangeSets, SegmentOutput { progress: Finished, pruned: 16665479, checkpoint: Some(SegmentOutputCheckpoint { block_number: Some(29669070), tx_number: None }) })] } stats=[PrunedSegmentInfo { segment: MerkleChangeSets, pruned: 16665479, progress: Finished }]
As my investigation, when we start new node, it automatically run this code path:
reth/crates/node/builder/src/launch/engine.rs
Line 157 in b48c72f
and there is prune run inside:
pub fn move_to_static_files(&self) -> RethResult<()> {
// Copies data from database to static files
let lowest_static_file_height =
self.static_file_producer.lock().copy_to_static_files()?.min_block_num();
// Deletes data which has been copied to static files.
if let Some(prune_tip) = lowest_static_file_height {
// Run the pruner so we don't potentially end up with higher height in the database vs
// static files during a pipeline unwind
let mut pruner = PrunerBuilder::new(Default::default())
.delete_limit(usize::MAX)
.build_with_provider_factory(self.provider_factory.clone());
pruner.run(prune_tip)?;
}
Ok(())
}
The merkle changesets need to prune older than 10k blocks here so that why it slow at start sequencer node
And look like this code is legacy, because it only related to receipts, but inside prune function (with Default mode is none segment on receipts), I don't see it prune anything except Merkle changesets 🤔
Need team take deeper look to review this
Steps to reproduce
we use 71c1247 then downgrade to bac0e1f then upgrade again to 71c1247 which is the prune slow event
Node logs
... log start nodes
2025-12-03 20:38:04.270 2025-12-03T13:38:04.143524597Z stdout F 2025-12-03T13:38:04.143492Z INFO reth::cli: RPC auth server started url=0.0.0.0:8551 ipc_endpoint=/tmp/reth.ipc
2025-12-03 20:38:04.270 2025-12-03T13:38:04.143509203Z stdout F 2025-12-03T13:38:04.143482Z INFO reth::cli: RPC WS server started url=0.0.0.0:8546
2025-12-03 20:38:04.270 2025-12-03T13:38:04.1435028Z stdout F 2025-12-03T13:38:04.143473Z INFO reth::cli: RPC HTTP server started url=0.0.0.0:8545
2025-12-03 20:38:04.270 2025-12-03T13:38:04.143486168Z stdout F 2025-12-03T13:38:04.143461Z INFO reth::cli: RPC IPC server started path=/tmp/reth.ipc
2025-12-03 20:38:04.270 2025-12-03T13:38:04.143232613Z stdout F 2025-12-03T13:38:04.143204Z DEBUG reth_ipc::server: removed existing IPC endpoint file endpoint="/tmp/reth.ipc"
2025-12-03 20:38:04.270 2025-12-03T13:38:04.142785598Z stdout F 2025-12-03T13:38:04.142743Z DEBUG reth::cli: Installing debug rpc endpoint
2025-12-03 20:38:04.270 2025-12-03T13:38:04.142769041Z stdout F 2025-12-03T13:38:04.142736Z DEBUG reth::cli: Installing miner DA rpc endpoint
2025-12-03 20:38:04.270 2025-12-03T13:38:04.142746113Z stdout F 2025-12-03T13:38:04.142721Z DEBUG reth::cli: Installing debug payload witness rpc endpoint
2025-12-03 20:38:04.270 2025-12-03T13:38:04.142626708Z stdout F 2025-12-03T13:38:04.142592Z DEBUG reth::cli: Using RPC module config http=Some(All) ws=Some(All)
2025-12-03 20:38:04.270 2025-12-03T13:38:04.141731129Z stdout F 2025-12-03T13:38:04.141699Z INFO reth::cli: Engine API handler initialized
2025-12-03 20:38:04.270 2025-12-03T13:38:04.141633917Z stdout F 2025-12-03T13:38:04.141612Z INFO reth::cli: Consensus engine initialized
2025-12-03 20:38:04.270 2025-12-03T13:38:04.141429575Z stdout F 2025-12-03T13:38:04.141413Z DEBUG reth::cli: Reading JWT auth secret file user_path="/tmp/jwt.txt"
2025-12-03 20:38:04.270 2025-12-03T13:38:04.141399759Z stdout F 2025-12-03T13:38:04.141371Z INFO reth::cli: Pruner initialized prune_config=PruneConfig { block_interval: 5, segments: PruneModes { sender_recovery: None, transaction_lookup: None, receipts: None, account_history: None, storage_history: None, bodies_history: None, merkle_changesets: Distance(10064), receipts_log_filter: () } }
2025-12-03 20:38:04.270 2025-12-03T13:38:04.141352489Z stdout F 2025-12-03T13:38:04.141308Z DEBUG storage::db::mdbx: Commit total_duration=95.103283ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 1, gc_wallclock: 2088, audit: 0, write: 0, sync: 4037, ending: 104, whole: 6231, gc_cputime: 1006, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-12-03 20:38:04.270 2025-12-03T13:38:04.046231835Z stdout F 2025-12-03T13:38:04.046176Z DEBUG pruner: Pruner finished tip_block_number=29679134 elapsed=1149.889397891s deleted_entries=16665479 limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 18446744073709551615, deleted: 16665479 }), time_limit: None } output=PrunerOutput { progress: Finished, segments: [(MerkleChangeSets, SegmentOutput { progress: Finished, pruned: 16665479, checkpoint: Some(SegmentOutputCheckpoint { block_number: Some(29669070), tx_number: None }) })] } stats=[PrunedSegmentInfo { segment: MerkleChangeSets, pruned: 16665479, progress: Finished }]
2025-12-03 20:38:04.270 2025-12-03T13:38:04.04619881Z stdout F 2025-12-03T13:38:04.046147Z DEBUG pruner: Segment pruning finished segment=MerkleChangeSets purpose=User to_block=29669070 prune_mode=Distance(10064) segment_output.pruned=16665479
2025-12-03 20:37:55.756 2025-12-03T13:37:55.539281657Z stdout F 2025-12-03T13:37:55.539225Z DEBUG storage::db::mdbx: Commit total_duration=2.627µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=true
2025-12-03 20:37:25.697 2025-12-03T13:37:25.538931703Z stdout F 2025-12-03T13:37:25.538832Z DEBUG storage::db::mdbx: Commit total_duration=3.282µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=true
2025-12-03 20:19:01.965 2025-12-03T13:18:54.504627975Z stdout F 2025-12-03T13:18:54.504587Z INFO txpool: Successfully reinserted local transactions from file txs_file="/db/l2_data/txpool-transactions-backup.rlp" num_txs=2108
2025-12-03 20:19:01.965 2025-12-03T13:18:54.157694811Z stdout F 2025-12-03T13:18:54.157651Z DEBUG txpool: removing stale transactions count=0
2025-12-03 20:19:01.965 2025-12-03T13:18:54.156812224Z stdout F 2025-12-03T13:18:54.156786Z DEBUG pruner: Segment pruning started segment=MerkleChangeSets purpose=User to_block=29669070 prune_mode=Distance(10064)
2025-12-03 20:19:01.965 2025-12-03T13:18:54.156790572Z stdout F 2025-12-03T13:18:54.156768Z DEBUG pruner: Pruner started tip_block_number=29679134
2025-12-03 20:19:01.965 2025-12-03T13:18:54.156589934Z stdout F 2025-12-03T13:18:54.156576Z INFO reth::cli: StaticFileProducer initialized
2025-12-03 20:19:01.965 2025-12-03T13:18:54.156555167Z stdout F 2025-12-03T13:18:54.156540Z DEBUG reth::cli: calling on_component_initialized hook
2025-12-03 20:19:01.965 2025-12-03T13:18:54.156519008Z stdout F 2025-12-03T13:18:54.156460Z DEBUG txpool: Check local persistent storage for saved transactions txs_file="/db/l2_data/txpool-transactions-backup.rlp"
2025-12-03 20:19:01.965 2025-12-03T13:18:54.156423924Z stdout F 2025-12-03T13:18:54.156413Z INFO rise::node: Spawned txpool maintenance task
2025-12-03 20:19:01.965 2025-12-03T13:18:54.156418971Z stdout F 2025-12-03T13:18:54.156393Z INFO rise::node: Transaction pool initialized
2025-12-03 20:19:01.965 2025-12-03T13:18:54.152251813Z stdout F 2025-12-03T13:18:54.152227Z DEBUG reth::cli: creating components
2025-12-03 20:19:01.965 2025-12-03T13:18:54.128230481Z stdout F 2025-12-03T13:18:54.128207Z DEBUG reth::cli: Spawning stages metrics listener task
2025-12-03 20:19:01.965 2025-12-03T13:18:54.128225704Z stdout F - Isthmus @1751968800
2025-12-03 20:19:01.965 2025-12-03T13:18:54.12822461Z stdout F - Holocene @1750845600
2025-12-03 20:19:01.965 2025-12-03T13:18:54.128223278Z stdout F - Granite @0
2025-12-03 20:19:01.965 2025-12-03T13:18:54.128221991Z stdout F - Fjord @0
2025-12-03 20:19:01.965 2025-12-03T13:18:54.128220824Z stdout F - Ecotone @0
2025-12-03 20:19:01.965 2025-12-03T13:18:54.128219762Z stdout F - Canyon @0
2025-12-03 20:19:01.965 2025-12-03T13:18:54.128218429Z stdout F - Regolith @0
2025-12-03 20:19:01.965 2025-12-03T13:18:54.128217326Z stdout F Post-merge hard forks (timestamp based):
2025-12-03 20:19:01.965 2025-12-03T13:18:54.128216205Z stdout F - Bedrock @0
2025-12-03 20:19:01.965 2025-12-03T13:18:54.12821482Z stdout F Pre-merge hard forks (block based):
2025-12-03 20:19:01.965 2025-12-03T13:18:54.128211146Z stdout F 2025-12-03T13:18:54.128192Z INFO reth::cli:
2025-12-03 20:19:01.965 2025-12-03T13:18:54.128200525Z stdout F 2025-12-03T13:18:54.128177Z DEBUG reth_db_common::init: Genesis already written, skipping.
2025-12-03 20:19:01.965 2025-12-03T13:18:54.126887497Z stdout F 2025-12-03T13:18:54.126866Z DEBUG reth::cli: Initializing genesis chain=11155931 genesis=0x3e679422992f508295d1f143e78f24d0ecd67d3228133085491842ba816eecc3
2025-12-03 20:19:01.965 2025-12-03T13:18:54.126850854Z stdout F 2025-12-03T13:18:54.126824Z INFO reth::cli: Starting metrics endpoint at 0.0.0.0:9001
2025-12-03 20:19:01.965 2025-12-03T13:18:54.126829757Z stdout F 2025-12-03T13:18:54.126779Z INFO reth::cli: Database opened
2025-12-03 20:19:01.965 2025-12-03T13:18:54.053142458Z stdout F 2025-12-03T13:18:54.053104Z INFO reth::cli: Verifying storage consistency.
2025-12-03 20:19:01.965 2025-12-03T13:18:53.949512514Z stdout F 2025-12-03T13:18:53.949477Z INFO reth::cli: Configuration loaded path="/db/l2_data/reth.toml"
2025-12-03 20:19:01.965 2025-12-03T13:18:53.948163352Z stdout F 2025-12-03T13:18:53.948120Z DEBUG reth_node_builder::launch::common: Raised file descriptor limit from=1048576 to=1048576
2025-12-03 20:19:01.964 2025-12-03T13:18:53.946011962Z stdout F 2025-12-03T13:18:53.945729Z DEBUG storage::db::mdbx: Commit total_duration=6.882043ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 374, audit: 0, write: 0, sync: 75, ending: 0, whole: 450, gc_cputime: 12, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-12-03 20:19:01.964 2025-12-03T13:18:53.870194112Z stdout F 2025-12-03T13:18:53.870175Z INFO reth::cli: Opening database path="/db/l2_data/db"
2025-12-03 20:19:01.964 2025-12-03T13:18:53.870186065Z stdout F 2025-12-03T13:18:53.870151Z INFO reth::cli: Starting reth version="1.8.3-dev (71c1247)"
2025-12-03 20:19:01.964 2025-12-03T13:18:53.869493372Z stdout F 2025-12-03T13:18:53.869441Z INFO reth::cli: Initialized tracing, debug log directory: /root/.cache/reth/logs/11155931
Platform(s)
Linux (x86)
Container Type
Kubernetes
What version/commit are you on?
What database version are you on?
2
Which chain / network are you on?
rise testnet sepolia
What type of node are you running?
Archive (default)
What prune config do you use, if any?
No response
If you've built Reth from source, provide the full command you used
No response
Code of Conduct
- I agree to follow the Code of Conduct