OOM on 1.9.3 (27a8c0f) Post Fusaka

[keccakk]

Describe the bug

Had an OOM post Fusaka, memory seemed to spike up suddenly after a bad block. Reth restarted after about 4 minutes later.

Steps to reproduce

[Service]
User=reth
Group=reth
Type=simple
Restart=always
RestartSec=5
ExecStart=reth node \
  --datadir /xxxxx/mainnet/reth \
  --authrpc.port 8551 \
  --authrpc.jwtsecret /xxxxx/ \
  --log.file.directory /xxxxx/mainnet/reth \
  --discovery.port 30303 \
  --port 30303 \
  --metrics 127.0.0.1:9001 \
  --http \
  --http.addr 0.0.0.0 \
  --http.port 8545 \
  --http.api "eth,net,trace,txpool,web3,rpc" \
  --http.corsdomain "XXXXX,XXXXX"

Node logs

Dec 03 17:22:14 arch reth[2585726]: 2025-12-03T22:22:14.750100Z  INFO Block added to canonical chain number=23935838 hash=0x6b5bfe7e65085295b7a1b74ae61cda2fa2d27646ea7519df81bf038106e3bdbb peers=131 txs=233 gas_used=30.47Mgas gas_throughput=58.64Mgas/second gas_limit=60.00Mgas full=50.8% base_fee=0.02Gwei blobs=7 excess_blobs=250 elapsed=519.643546ms
Dec 03 17:22:14 arch reth[2585726]: 2025-12-03T22:22:14.919121Z  INFO Canonical chain committed number=23935838 hash=0x6b5bfe7e65085295b7a1b74ae61cda2fa2d27646ea7519df81bf038106e3bdbb elapsed=806.2µs
Dec 03 17:22:23 arch reth[2585726]: 2025-12-03T22:22:23.740790Z  INFO Status connected_peers=131 latest_block=23935838
Dec 03 17:22:25 arch reth[2585726]: 2025-12-03T22:22:25.457464Z  INFO Received block from consensus engine number=23935839 hash=0x3b28163fd808f8febb46b220b7bf588e0ac1039f22ac81a7c3c017f622bd15b8
Dec 03 17:22:25 arch reth[2585726]: 2025-12-03T22:22:25.533942Z  WARN Failed to validate header 0x3b28163fd808f8febb46b220b7bf588e0ac1039f22ac81a7c3c017f622bd15b8 against parent: invalid excess blob gas: got 32942754, expected 33117516; parent excess blob gas: 32811682, parent blob gas used: 917504 block=RecoveredBlock { block: SealedBlock { header: SealedHeader { hash: OnceLock(0x3b28163fd808f8febb46b220b7bf588e0>
Dec 03 17:22:25 arch reth[2585726]: 6cf091, 0x960b7ebf0bb53c7aa742986d3144143a99b547c9075f1afccfd7bc6e326cf092, 0xbfd2cfff8855c931101af72e6b5adcc13b26b884b98f7e1665e8afb1935164a1, 0xe66f835ef1511d29bcdd6fc432fb7aeefe364f1708b5cb78bb1dd4d25e4833d4] }]), input: 0x12aa3caf0000000000000000000000000004d60f031e000107004585fe77225b41b697c938b018e2ac67ac5a20c00044128acb08000000000000000000000000d36f184111225d065884e129566>
Dec 03 17:22:25 arch reth[2585726]: 0000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000003a225ff5f420faf56e3b9d6e0900000000000000015455c918e405a2831fbff8595c0aae35ee3db9d100000000000000000000000000000000000000000000000000000000000000800000000000000000000000006>
Dec 03 17:22:25 arch reth[2585726]: 000000000000000000000000000000000000000000000000000000000000000000002887b22536f75726365223a226c692e6669222c22416d6f756e74496e555344223a2232342e373635313339353331343732333534222c22416d6f756e744f7574555344223a2232342e393030353233393636393439383837222c22526566657272616c223a22222c22466c616773223a302c22416d6f756e744f7574223a2231313531313536363532333533373330303230383236222c2254696d65>
Dec 03 17:22:25 arch reth[2585726]: 7563926639922241532429819911437201, s: 56465499571647771781405502136090427531776215388036603515599035950417423856042 }, hash: OnceLock(<uninit>) }), Eip2930(Signed { tx: TxEip2930 { chain_id: 1, nonce: 4324266, gas_price: 66695960, gas_limit: 500000, to: Call(0xb1f89123c8e5bd15b2aac12f55d82cbf38345974), value: 12256580000000000, access_list: AccessList([]), input: 0x }, signatur>
Dec 03 17:22:25 arch reth[2585726]: 36 }, hash: OnceLock(<uninit>) }), Eip2930(Signed { tx: TxEip2930 { chain_id: 1, nonce: 4324361, gas_price: 59826855, gas_limit: 500000, to: Call(0xc4f1352ac210b8551a14c9c4c173460943c2f73f), value: 10991960000000000, access_list: AccessList([]), input: 0x }, signature: Signature { y_parity: true, r: 90003898909402786077084897247764535769234250039946682817757271321759316710652, s>
Dec 03 17:22:25 arch reth[2585726]: 7b6cbdc63d2fb801e9f61f42c287d05e9a032e34c000000000000000000000000000000000054454d5030353030312d3500000000015957759a47f657c3b8f0b93cdbe75ad6140b7042d47f80260f0523a5ed22a5000000000000000000000000000000000054454d5030353030322d350000000001d3baaa1f6478cee050d24e15b0589b565ab70c4ad6bda5d49d1da58da01017000000000000000000000000000000000054454d5030353030332d350000000004ca6649cbca137cea07>
Dec 03 17:22:25 arch reth[2585726]: 000000000000000000000000000000000008fe8a40000000000000000000000000000000000415242322d3800000000000000000000000000000000000000000000000000000000000000000000000000008fe8a4000000000000000000000000000000000041544f4d2d380000000000000000000000000000000000000000000000000000000000000000000000000006106b22000000000000000000000000000000000041544f4d322d38000000000000000000000000000000000000>
Dec 03 17:22:25 arch reth[2585726]: 0000000000054454d5030383031352d3800000000000000000000000000000000000000000000000000000000800000000010f852000000000000000000000000000000000054454d5030383031362d3800000000000000000000000000000000000000000000000000000000800000000010f852000000000000000000000000000000000054454d5030383031372d3800000000000000000000000000000000000000000000000000000000800000000010f85200000000000000000000>
Dec 03 17:22:25 arch reth[2585726]: 000000009742dd76d59d0126ab50000114bfbe1331b6e81 }, signature: Signature { y_parity: true, r: 11154418555959077016783757278161942749407944354919476574993022655799437683848, s: 48456368501396695840195591318452331796222797374434315361734623674686341430281 }, hash: OnceLock(<uninit>) }), Eip1559(Signed { tx: TxEip1559 { chain_id: 1, nonce: 94921, gas_limit: 80803, max_fee_per_gas: 2>
Dec 03 17:22:25 arch reth[2585726]: Eip1559(Signed { tx: TxEip1559 { chain_id: 1, nonce: 495, gas_limit: 190616, max_fee_per_gas: 27701337, max_priority_fee_per_gas: 27701337, to: Call(0x66a9893cc07d91d95644aedd05d03f95e1dba8af), value: 0, access_list: AccessList([]), input: 0x3593564c000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000a0000>
Dec 03 17:22:25 arch reth[2585726]: ned { tx: TxLegacy { chain_id: Some(1), nonce: 12, gas_price: 22883114, gas_limit: 91008, to: Call(0xdac17f958d2ee523a2206206994597c13d831ec7), value: 0, input: 0xa9059cbb000000000000000000000000708c218d548c524d801f0137d16a2121048ddb180000000000000000000000000000000000000000000000000000000000001388 }, signature: Signature { y_parity: false, r: 73088093242269909114809257083045734>
Dec 03 17:22:25 arch reth[2585726]: 595495200387046983517265 }, hash: OnceLock(<uninit>) }), Legacy(Signed { tx: TxLegacy { chain_id: Some(1), nonce: 1, gas_price: 22011376, gas_limit: 21000, to: Call(0xd59df0bd79d39b45324c033eab6a323f44d32b5e), value: 169515569000, input: 0x }, signature: Signature { y_parity: true, r: 100391635375663370633849946315604822416263709603721751145865659702717283438541, s: 203565024247>
Dec 03 17:22:25 arch reth[2585726]: 0000000000000000000000000000000000297fbe1f4da7613c79982e0decc48fbb56101fc919bc341d84e7afdecf2d4f007e017c00000000000000000000000000000000000000000000000000000000000002f5000000000000000000000000000000000000000000000000000000000000002a4b0f546f19b57b53c362a3957c398a6cb0dd4e208ab83caff0a99fe53b0a42000000000000000000000000000000000000000000000000000000000000000120d0c561e22208c1d2a2cc5>
Dec 03 17:22:25 arch reth[2585726]: 74c33c6bf6fbcd38baab813cbb6, amount: 17775650 }, Withdrawal { index: 110414044, validator_index: 1581395, address: 0xd007058e9b58e74c33c6bf6fbcd38baab813cbb6, amount: 17745062 }, Withdrawal { index: 110414045, validator_index: 1581396, address: 0xd007058e9b58e74c33c6bf6fbcd38baab813cbb6, amount: 63784913 }, Withdrawal { index: 110414046, validator_index: 1581397, address: 0xd007>
Dec 03 17:22:25 arch reth[2585726]: 2025-12-03T22:22:25.537997Z  WARN Invalid block error on new payload invalid_hash=0x3b28163fd808f8febb46b220b7bf588e0ac1039f22ac81a7c3c017f622bd15b8 invalid_number=23935839 validation_err=invalid excess blob gas: got 32942754, expected 33117516; parent excess blob gas: 32811682, parent blob gas used: 917504
Dec 03 17:22:25 arch reth[2585726]: 2025-12-03T22:22:25.538034Z  WARN Bad block with hash invalid_ancestor=BlockWithParent { parent: 0x6b5bfe7e65085295b7a1b74ae61cda2fa2d27646ea7519df81bf038106e3bdbb, block: NumHash { number: 23935839, hash: 0x3b28163fd808f8febb46b220b7bf588e0ac1039f22ac81a7c3c017f622bd15b8 } }
Dec 03 17:22:25 arch reth[2585726]: 2025-12-03T22:22:25.538234Z  WARN Encountered invalid block number=23935839 hash=0x3b28163fd808f8febb46b220b7bf588e0ac1039f22ac81a7c3c017f622bd15b8
Dec 03 17:22:38 arch reth[2585726]: 2025-12-03T22:22:38.981962Z  INFO Received block from consensus engine number=23935839 hash=0xcc821eba3d41c44d8bdfacfd1740563042e33b9c5393779d532678c6f7c2580a
Dec 03 17:22:39 arch reth[2585726]: 2025-12-03T22:22:39.655880Z  INFO Regular root task finished regular_state_root=0x0bd2a7f14f73d8f3222b8062d30bddc18c11bbbf80fcb470a95f1b205ef80248 elapsed=563.396989ms
.....
Dec 03 17:25:24 arch reth[2585726]: 2025-12-03T22:25:24.972921Z  INFO Regular root task finished regular_state_root=0xa80b8189f78de3eb38f0451922f11f10ff5eb63646ae929a6bceeb4e76e78e07 elapsed=254.829966ms
Dec 03 17:25:24 arch reth[2585726]: 2025-12-03T22:25:24.973771Z  INFO Block added to canonical chain number=23935853 hash=0x184f36277a5f07d7628e92bed2eb801127371aefb1a625f0ff4ad8d7f0ed4a1e peers=131 txs=217 gas_used=30.90Mgas gas_throughput=103.03Mgas/second gas_limit=60.00Mgas full=51.5% base_fee=0.02Gwei blobs=0 excess_blobs=271 elapsed=299.942668ms
Dec 03 17:25:25 arch reth[2585726]: 2025-12-03T22:25:25.158979Z  INFO Canonical chain committed number=23935853 hash=0x184f36277a5f07d7628e92bed2eb801127371aefb1a625f0ff4ad8d7f0ed4a1e elapsed=400.299µs
Dec 03 17:25:40 arch reth[2585726]: 2025-12-03T22:25:40.029307Z  INFO Received block from consensus engine number=23935809 hash=0xd6cbb630c84c44396a2360d271a9f9f0c8b4c3bc13a442b6c418ad3bea8b8cc4
Dec 03 17:26:07 arch systemd[1]: reth.service: A process of this unit has been killed by the OOM killer.
Dec 03 17:26:10 arch systemd[1]: reth.service: Main process exited, code=killed, status=9/KILL
Dec 03 17:26:10 arch systemd[1]: reth.service: Failed with result 'oom-kill'.
Dec 03 17:26:10 arch systemd[1]: reth.service: Consumed 14h 29min 15.151s CPU time, 20.2G memory peak.
Dec 03 17:26:15 arch systemd[1]: reth.service: Scheduled restart job, restart counter is at 4.
Dec 03 17:26:15 arch systemd[1]: Started Reth Execution Client.
Dec 03 17:26:15 arch reth[2673735]: 2025-12-03T22:26:15.571588Z  INFO Initialized tracing, debug log directory: xxxxx/mainnet/reth/mainnet
Dec 03 17:26:15 arch reth[2673735]: 2025-12-03T22:26:15.576096Z  INFO Starting reth version="1.9.3 (27a8c0f)"
Dec 03 17:26:15 arch reth[2673735]: 2025-12-03T22:26:15.576773Z  INFO Opening database path="xxxxx/mainnet/reth/db"
Dec 03 17:26:15 arch reth[2673735]: 2025-12-03T22:26:15.594376Z  INFO Launching node
Dec 03 17:26:15 arch reth[2673735]: 2025-12-03T22:26:15.604891Z  INFO Configuration loaded path="xxxxx/mainnet/reth/reth.toml"
Dec 03 17:26:15 arch reth[2673735]: 2025-12-03T22:26:15.630353Z  INFO Verifying storage consistency.
Dec 03 17:26:15 arch reth[2673735]: 2025-12-03T22:26:15.650703Z  INFO Database opened
Dec 03 17:26:15 arch reth[2673735]: 2025-12-03T22:26:15.651413Z  INFO Starting metrics endpoint at 127.0.0.1:9001
Dec 03 17:26:15 arch reth[2673735]: 2025-12-03T22:26:15.654441Z  INFO
Dec 03 17:26:15 arch reth[2673735]: Pre-merge hard forks (block based):
Dec 03 17:26:15 arch reth[2673735]: - Frontier                         @0
Dec 03 17:26:15 arch reth[2673735]: - Homestead                        @1150000
Dec 03 17:26:15 arch reth[2673735]: - Dao                              @1920000
Dec 03 17:26:15 arch reth[2673735]: - Tangerine                        @2463000
Dec 03 17:26:15 arch reth[2673735]: - SpuriousDragon                   @2675000
Dec 03 17:26:15 arch reth[2673735]: - Byzantium                        @4370000
Dec 03 17:26:15 arch reth[2673735]: - Constantinople                   @7280000
Dec 03 17:26:15 arch reth[2673735]: - Petersburg                       @7280000
Dec 03 17:26:15 arch reth[2673735]: - Istanbul                         @9069000
Dec 03 17:26:15 arch reth[2673735]: - MuirGlacier                      @9200000
Dec 03 17:26:15 arch reth[2673735]: - Berlin                           @12244000
Dec 03 17:26:15 arch reth[2673735]: - London                           @12965000
Dec 03 17:26:15 arch reth[2673735]: - ArrowGlacier                     @13773000
Dec 03 17:26:15 arch reth[2673735]: - GrayGlacier                      @15050000
Dec 03 17:26:15 arch reth[2673735]: Merge hard forks:
Dec 03 17:26:15 arch reth[2673735]: - Paris                            @58750000000000000000000 (network is known to be merged)
Dec 03 17:26:15 arch reth[2673735]: Post-merge hard forks (timestamp based):
Dec 03 17:26:15 arch reth[2673735]: - Shanghai                         @1681338455
Dec 03 17:26:15 arch reth[2673735]: - Cancun                           @1710338135          blob: (target: 3, max: 6, fraction: 3338477)
Dec 03 17:26:15 arch reth[2673735]: - Prague                           @1746612311          blob: (target: 6, max: 9, fraction: 5007716)
Dec 03 17:26:15 arch reth[2673735]: - Osaka                            @1764798551          blob: (target: 6, max: 9, fraction: 5007716)
Dec 03 17:26:15 arch reth[2673735]: - Bpo1                             @1765290071          blob: (target: 10, max: 15, fraction: 8346193)
Dec 03 17:26:15 arch reth[2673735]: - Bpo2                             @1767747671          blob: (target: 14, max: 21, fraction: 11684671)
Dec 03 17:26:15 arch reth[2673735]: 2025-12-03T22:26:15.664527Z  INFO Transaction pool initialized

Platform(s)

Linux (x86)

Container Type

Not running in a container

What version/commit are you on?

Reth Version: 1.9.3
Commit SHA: 27a8c0f5a6dfb27dea84c5751776ecabdd069646
Build Timestamp: 2025-11-20T15:10:06.385593747Z
Build Features: asm_keccak,jemalloc,otlp
Build Profile: maxperf

What database version are you on?

Available on request.

Which chain / network are you on?

Mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

None

If you've built Reth from source, provide the full command you used

RUSTFLAGS="-C target-cpu=native" cargo build --profile maxperf

Code of Conduct

• I agree to follow the Code of Conduct

[Rjected]

do you have grafana set up? if so can you share an image of the jemalloc memory panel from the reth dashboard?

[keccakk]

[lightclient]

I'm also seeing OOMs on one of my nodes.

$ reth --version
Reth Version: 1.9.3
Commit SHA: 27a8c0f5a6dfb27dea84c5751776ecabdd069646
Build Timestamp: 2025-12-03T16:53:51.877017378Z
Build Features: asm_keccak,jemalloc,min_debug_logs,otlp
Build Profile: release

[frisitano]

We are also experiencing memory issues on reth 1.9.1, which have worsened after Fusaka

[piotruslalka]

I also had a OOM and am continuing to receive 40-50 seconds of hang time while Reth is using 100% processing power.

[Wed Dec  3 22:26:07 2025] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/reth.service,task=reth,pid=816,uid=1001

[Wed Dec  3 22:26:07 2025] Out of memory: Killed process 816 (reth) total-vm:9162744776kB, anon-rss:25175924kB, file-rss:7792kB, shmem-rss:0kB, UID:1001 pgtables:1635572kB oom_score_adj:0

Also this from the RETH logs:

Dec 04 09:16:13 myserver reth[807]: 2025-12-04T09:16:13.469425Z  WARN The database read transaction has been open for too long. Backtrace:
Dec 04 09:16:13 myserver reth[807]:    0: <unknown>
Dec 04 09:16:13 myserver reth[807]:    1: <unknown>
Dec 04 09:16:13 myserver reth[807]:    2: <unknown>
Dec 04 09:16:13 myserver reth[807]:    3: <unknown>
Dec 04 09:16:13 myserver reth[807]:    4: <unknown>
Dec 04 09:16:13 myserver reth[807]:    5: <unknown>
Dec 04 09:16:13 myserver reth[807]:    6: <unknown>
Dec 04 09:16:13 myserver reth[807]:    7: <unknown>
Dec 04 09:16:13 myserver reth[807]:    8: <unknown>
Dec 04 09:16:13 myserver reth[807]:    9: <unknown>
Dec 04 09:16:13 myserver reth[807]:   10: <unknown>
Dec 04 09:16:13 myserver reth[807]:   11: <unknown>
Dec 04 09:16:13 myserver reth[807]:   12: <unknown>
Dec 04 09:16:13 myserver reth[807]:   13: <unknown>
Dec 04 09:16:13 myserver reth[807]:  open_duration=108.08704776s self.txn_id=21158333
Dec 04 09:16:13 myserver reth[807]: 2025-12-04T09:16:13.487221Z  INFO Regular root task finished regular_state_root=0xd738b98cccc4b9e9668e7c323aa61866fa6cedd3d2fdd3b663eac2a546c94ab4 elapsed=110.1244991
89s
Dec 04 09:16:13 myserver reth[807]: 2025-12-04T09:16:13.488997Z  WARN The database read transaction has been open for too long. Backtrace:
Dec 04 09:16:13 myserver reth[807]:    0: <unknown>
Dec 04 09:16:13 myserver reth[807]:    1: <unknown>
Dec 04 09:16:13 myserver reth[807]:    2: <unknown>
Dec 04 09:16:13 myserver reth[807]:    3: <unknown>
Dec 04 09:16:13 myserver reth[807]:    4: <unknown>
Dec 04 09:16:13 myserver reth[807]:    5: <unknown>
Dec 04 09:16:13 myserver reth[807]:    6: <unknown>
Dec 04 09:16:13 myserver reth[807]:    7: <unknown>
Dec 04 09:16:13 myserver reth[807]:    8: <unknown>
Dec 04 09:16:13 myserver reth[807]:    9: <unknown>
Dec 04 09:16:13 myserver reth[807]:   10: <unknown>
Dec 04 09:16:13 myserver reth[807]:   11: <unknown>
Dec 04 09:16:13 myserver reth[807]:  open_duration=110.394767648s self.txn_id=21158333
Dec 04 09:16:13 myserver reth[807]: 2025-12-04T09:16:13.490199Z  INFO Block added to fork chain number=23938709 hash=0xa9dc6b5070304c4989be8114a6d6a6816cb15679948fd82f2db6e300a73220f8 elapsed=110.395935
01s
Dec 04 09:16:13 myserver reth[807]: 2025-12-04T09:16:13.490566Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0xa9dc6b5070304c4989be8114a6d6a6816cb15679948fd82f2d
b6e300a73220f8) })
Dec 04 09:16:16 myserver reth[807]: 2025-12-04T09:16:16.455221Z  INFO Received block from consensus engine number=23938752 hash=0x08cdcad9436ce2d0843fe1ad1eb9caaa04d37beab44b0a9c0e16893eb343156b
Dec 04 09:16:17 myserver reth[807]: 2025-12-04T09:16:17.000801Z  INFO Regular root task finished regular_state_root=0xdc77658f0462f3b54f59661c3e2746599bd76225cc3939b53b4b875324495b02 elapsed=439.869308m
s

Not sure how I can help, but just wanted to add a datapoint.

[aborigeth]

I also had a OOM and am continuing to receive 40-50 seconds of hang time while Reth is using 100% processing power.

[Wed Dec 3 22:26:07 2025] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/reth.service,task=reth,pid=816,uid=1001

[Wed Dec 3 22:26:07 2025] Out of memory: Killed process 816 (reth) total-vm:9162744776kB, anon-rss:25175924kB, file-rss:7792kB, shmem-rss:0kB, UID:1001 pgtables:1635572kB oom_score_adj:0

Also this from the RETH logs:

Dec 04 09:16:13 myserver reth[807]: 2025-12-04T09:16:13.469425Z WARN The database read transaction has been open for too long. Backtrace:
Dec 04 09:16:13 myserver reth[807]: 0:
Dec 04 09:16:13 myserver reth[807]: 1:
Dec 04 09:16:13 myserver reth[807]: 2:
Dec 04 09:16:13 myserver reth[807]: 3:
Dec 04 09:16:13 myserver reth[807]: 4:
Dec 04 09:16:13 myserver reth[807]: 5:
Dec 04 09:16:13 myserver reth[807]: 6:
Dec 04 09:16:13 myserver reth[807]: 7:
Dec 04 09:16:13 myserver reth[807]: 8:
Dec 04 09:16:13 myserver reth[807]: 9:
Dec 04 09:16:13 myserver reth[807]: 10:
Dec 04 09:16:13 myserver reth[807]: 11:
Dec 04 09:16:13 myserver reth[807]: 12:
Dec 04 09:16:13 myserver reth[807]: 13:
Dec 04 09:16:13 myserver reth[807]: open_duration=108.08704776s self.txn_id=21158333
Dec 04 09:16:13 myserver reth[807]: 2025-12-04T09:16:13.487221Z INFO Regular root task finished regular_state_root=0xd738b98cccc4b9e9668e7c323aa61866fa6cedd3d2fdd3b663eac2a546c94ab4 elapsed=110.1244991
89s
Dec 04 09:16:13 myserver reth[807]: 2025-12-04T09:16:13.488997Z WARN The database read transaction has been open for too long. Backtrace:
Dec 04 09:16:13 myserver reth[807]: 0:
Dec 04 09:16:13 myserver reth[807]: 1:
Dec 04 09:16:13 myserver reth[807]: 2:
Dec 04 09:16:13 myserver reth[807]: 3:
Dec 04 09:16:13 myserver reth[807]: 4:
Dec 04 09:16:13 myserver reth[807]: 5:
Dec 04 09:16:13 myserver reth[807]: 6:
Dec 04 09:16:13 myserver reth[807]: 7:
Dec 04 09:16:13 myserver reth[807]: 8:
Dec 04 09:16:13 myserver reth[807]: 9:
Dec 04 09:16:13 myserver reth[807]: 10:
Dec 04 09:16:13 myserver reth[807]: 11:
Dec 04 09:16:13 myserver reth[807]: open_duration=110.394767648s self.txn_id=21158333
Dec 04 09:16:13 myserver reth[807]: 2025-12-04T09:16:13.490199Z INFO Block added to fork chain number=23938709 hash=0xa9dc6b5070304c4989be8114a6d6a6816cb15679948fd82f2db6e300a73220f8 elapsed=110.395935
01s
Dec 04 09:16:13 myserver reth[807]: 2025-12-04T09:16:13.490566Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0xa9dc6b5070304c4989be8114a6d6a6816cb15679948fd82f2d
b6e300a73220f8) })
Dec 04 09:16:16 myserver reth[807]: 2025-12-04T09:16:16.455221Z INFO Received block from consensus engine number=23938752 hash=0x08cdcad9436ce2d0843fe1ad1eb9caaa04d37beab44b0a9c0e16893eb343156b
Dec 04 09:16:17 myserver reth[807]: 2025-12-04T09:16:17.000801Z INFO Regular root task finished regular_state_root=0xdc77658f0462f3b54f59661c3e2746599bd76225cc3939b53b4b875324495b02 elapsed=439.869308m
s

Not sure how I can help, but just wanted to add a datapoint.

So far, reverting to the pre-fusaka fork block has been helpful for me.

[mattsse]

if anyone could provide debug logs that would be helpful

[frisitano]

if anyone could provide debug logs that would be helpful

2025-12-04T12:43:17.271410Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103123 next_block=19103127 expected=19103124 range_size=1
2025-12-04T12:43:17.310342Z  INFO reth::cli: Status connected_peers=17 latest_block=23939757
2025-12-04T12:43:17.370743Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19523938 next_block=19523940 expected=19523939 range_size=1
2025-12-04T12:43:17.676977Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19523944 next_block=19523948 expected=19523945 range_size=5
2025-12-04T12:43:17.696068Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103133 next_block=19103136 expected=19103134 range_size=7
2025-12-04T12:43:22.193333Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103136 next_block=19103139 expected=19103137 range_size=1
2025-12-04T12:43:22.278589Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103139 next_block=19103141 expected=19103140 range_size=1
2025-12-04T12:43:22.284998Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19523949 next_block=19523951 expected=19523950 range_size=2
2025-12-04T12:43:22.977316Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103141 next_block=19103144 expected=19103142 range_size=1
2025-12-04T12:43:23.470112Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103144 next_block=19103147 expected=19103145 range_size=1
2025-12-04T12:43:23.575604Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19523953 next_block=19523955 expected=19523954 range_size=3
2025-12-04T12:43:23.706627Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103147 next_block=19103150 expected=19103148 range_size=1
2025-12-04T12:43:23.993426Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103150 next_block=19103153 expected=19103151 range_size=1
2025-12-04T12:43:24.979268Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103153 next_block=19103156 expected=19103154 range_size=1
2025-12-04T12:43:25.178254Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103156 next_block=19103158 expected=19103157 range_size=1
2025-12-04T12:43:25.278621Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19523955 next_block=19523957 expected=19523956 range_size=1
2025-12-04T12:43:26.294300Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103158 next_block=19103160 expected=19103159 range_size=1
2025-12-04T12:43:26.315413Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19523960 next_block=19523962 expected=19523961 range_size=4
2025-12-04T12:43:26.906987Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103160 next_block=19103165 expected=19103161 range_size=1
2025-12-04T12:43:27.318830Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103165 next_block=19103168 expected=19103166 range_size=1
2025-12-04T12:43:27.672930Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103170 next_block=19103172 expected=19103171 range_size=3
2025-12-04T12:43:27.989280Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103172 next_block=19103174 expected=19103173 range_size=1
2025-12-04T12:43:28.802906Z DEBUG rpc::eth::filter: Non-consecutive block detected, stopping range collection last_block=19103176 next_block=19103179 expected=19103177 range_size=3
Killed

We use a filter in eth_getLogs. All eth_getLogs rpc requests are sequential - not concurrent. Just a guess, but I wonder if there is a memory leak due to a large number of tokio tasks being created from these queries (maybe in the rpc cache?). Tokio console would have been useful to validate/reject this suspicion, but I notice you don't use it.

[piotruslalka]

This was my debug:

Dec 04 19:03:53 myserver reth[86483]: 2025-12-04T19:03:53.981605Z  INFO reth::cli: Status connected_peers=71 latest_block=23941617
Dec 04 19:03:55 myserver reth[86483]: 2025-12-04T19:03:55.476210Z DEBUG txpool: [0x5ada125f274d841c275d4bd049352b1d9c63ffeaf5665bab957a197dfd572db0] storing blob sidecar
Dec 04 19:03:57 myserver reth[86483]: 2025-12-04T19:03:57.052161Z DEBUG txpool: [0xd8f9b34c796c90be6aaff1e758b285bf570364866ab2329138d85827054e470e] storing blob sidecar
Dec 04 19:03:58 myserver reth[86483]: 2025-12-04T19:03:58.486715Z DEBUG txpool: [0xd6cdfe97c177bab3d4e692312ff52e886d2e767259e03943bd26c56f83efecd0] storing blob sidecar
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.435500Z DEBUG engine::tree: received new engine message msg=Request(NewPayload(parent: 0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1
f7f3636d9, number: 23941585, hash: 0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa))
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.435581Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payloa
d:insert_block_or_payload: engine::tree: Inserting new block into tree block=NumHash { number: 23941585, hash: 0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa } parent=0x6b91530e4e0e8
9f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.435746Z DEBUG reth::cli: Event: Handler(BlockReceived(NumHash { number: 23941585, hash: 0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76
b6062372aa }))
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.435782Z  INFO reth_node_events::node: Received block from consensus engine number=23941585 hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.436184Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload: engine::tree: found canonical state for block in database, creating provider builder hash=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 number=23941584
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.436243Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}:state provider: engine::tree::payload_validator: found canonical state for block in database, creating provider builder hash=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 number=23941584
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.436291Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}:plan_state_root_computation: engine::tree::payload_validator: Planned state root computation strategy strategy=Parallel
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.436307Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}: engine::tree::payload_validator: Deciding which state root algorithm to run strategy=Parallel
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.436510Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}:spawn_payload_processor:spawn_cache_exclusive:cache_for{parent_hash=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9}: reth_engine_tree::tree::payload_processor: creating new execution cache on cache miss
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.436683Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}:execute_block: engine::tree::payload_validator: Executing block
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.678820Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}:execute_block: engine::tree::payload_validator: Executed block elapsed=242.058139ms
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.679130Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}:spawn_payload_processor:spawn_cache_exclusive:prewarm task:prewarm: engine::tree::prewarm: Terminating prewarm execution
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.749970Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}: engine::tree::payload_validator: Calculating block state root
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.750012Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}: engine::tree::payload_validator: Using parallel state root algorithm
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.750031Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}:compute_state_root_parallel:compute_trie_input: engine::tree::payload_validator: Parent found on disk
Dec 04 19:04:02 myserver reth[86483]: 2025-12-04T19:04:02.751462Z DEBUG on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payload:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}:compute_state_root_parallel: trie::parallel_state_root: pre-calculating storage roots len=1533
Dec 04 19:04:19 myserver reth[86483]: 2025-12-04T19:04:18.995768Z DEBUG net::session: timed out outgoing request id=562 remote_peer_id=0x2d4f8615679831e73a765a04d72e077b267b8a3faa64cfb719189690e988f24968b59eb9473d774bfea3285eb114b24ead2f7600c0477fb211194044c6f37473
Dec 04 19:04:19 myserver reth[86483]: 2025-12-04T19:04:19.009321Z DEBUG net::session: timed out outgoing request id=2861 remote_peer_id=0x54354d9b6852f426c76f26a25656f82bf9d2e63dac931b68623c24ed58bcc460feae69258064991e01f388f85b0932634ac75f95d6d82c17f72fc54c12f9ecaa
Dec 04 19:04:19 myserver reth[86483]: 2025-12-04T19:04:19.251574Z DEBUG net::session: timed out outgoing request id=45 remote_peer_id=0xa380dae8496339e483042fb739d3b83f43370f7f69252b12e0ebe5b3c5c0cde8fe7b15c0c744e1cb2c964ca93185a0cc5757d44110572d179d6647e3256078f3
Dec 04 19:04:22 myserver reth[86483]: 2025-12-04T19:04:22.002572Z DEBUG providers::state::overlay: Reverted to target block block_hash=Some(0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9) from_block=23941584 num_trie_updates=81486 num_state_updates=27350
Dec 04 19:04:22 myserver reth[86483]: 2025-12-04T19:04:22.321448Z DEBUG providers::state::overlay: Reverted to target block block_hash=Some(0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9) from_block=23941584 num_trie_updates=81486 num_state_updates=27350
Dec 04 19:04:22 myserver reth[86483]: 2025-12-04T19:04:22.470188Z DEBUG providers::state::overlay: Reverted to target block block_hash=Some(0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9) from_block=23941584 num_trie_updates=81486 num_state_updates=27350

Then it keeps repeating "Reverted to target block" for over a whole minute while the CPU is 100% and RAM spikes to max.

Eventually, we get the "The database read transactino has been open for too long" message:

Dec 04 19:05:35 myserver reth[86483]: 2025-12-04T19:05:35.192851Z  WARN on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payloa
d:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}:compute_state_root_parallel: storage::db::mdbx: The dat
abase read transaction has been open for too long. Backtrace:
Dec 04 19:05:35 myserver reth[86483]:    0: <unknown>
Dec 04 19:05:35 myserver reth[86483]:    1: <unknown>
Dec 04 19:05:35 myserver reth[86483]:    2: <unknown>
Dec 04 19:05:35 myserver reth[86483]:    3: <unknown>
Dec 04 19:05:35 myserver reth[86483]:    4: <unknown>
Dec 04 19:05:35 myserver reth[86483]:    5: <unknown>
Dec 04 19:05:35 myserver reth[86483]:    6: <unknown>
Dec 04 19:05:35 myserver reth[86483]:    7: <unknown>
Dec 04 19:05:35 myserver reth[86483]:    8: <unknown>
Dec 04 19:05:35 myserver reth[86483]:    9: <unknown>
Dec 04 19:05:35 myserver reth[86483]:   10: <unknown>
Dec 04 19:05:35 myserver reth[86483]:   11: <unknown>
Dec 04 19:05:35 myserver reth[86483]:   12: <unknown>
Dec 04 19:05:35 myserver reth[86483]:   13: <unknown>
Dec 04 19:05:35 myserver reth[86483]:  open_duration=86.811757763s self.txn_id=21209254
Dec 04 19:05:35 myserver reth[86483]: 2025-12-04T19:05:35.223348Z  INFO on_new_payload{block_hash=0x18fc73379e58de4cab668cae31f326358dcab01d7166d45ef85b76b6062372aa block_num=23941585}:try_insert_payloa
d:insert_block_or_payload:validate_block_with_state{parent=0x6b91530e4e0e89f728564f31e2716ab9944881537c46b1781710bb1f7f3636d9 type_name="payload"}: engine::tree::payload_validator: Regular root task fin
ished regular_state_root=0xce60aedb8c797b8f9698068ad5ab62291a69caac6101aa2ac4ac0e6ece38c5e9 elapsed=92.473332219s

[mattsse]

could you also add some system info, core count, ram?

[keccakk]

Intel G7400 LGA1700
Dual Core - 4 threads
32GB RAM
8TB NVMe
Arch Linux 6.12.57-1-lts

Also, I didn't have any issues running Fusaka on Sepolia as recently as 3 weeks ago. While also running Mainnet at the same time. This only started after Fusaka on Mainnet.