State root mismatch on Base Mainnet caused by possible DB corruption

[meyer9]

Describe the bug

While testing Reth v1.11, we consistently got state root mismatches after a while of running. We tracked a few of these mismatches down to the source and they always occurred where a hash was not removed from a branch node compact when a leaf was deleted.

This repo has all the details of where exactly the database was corrupted: https://github.com/meyer9/reth-db-corruption

Steps to reproduce

Run Reth v1.11 with Base Mainnet.

Node logs

The only logs were just repeated:


State root task returned incorrect state root

Platform(s)

No response

Container Type

Kubernetes

What version/commit are you on?

v1.11.2 tag

What database version are you on?

2

Which chain / network are you on?

Base Mainnet

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[dhyaniarun1993]

@meyer9 I was facing the same issue which was fixed by bumping reth to v1.11.3 and running reth db --datadir <datadir> repair-trie

[meyer9]

Thanks for the info! Testing now with v1.11.3!

[meyer9]

Seems to be fixed with v1.11.3