history_by_block_hash returns invalid parent block state for replaying block

[Donavern]

Hello all, I currently am using exex on reth v1.8.2, for a new range of blocks (N ~ N + X), I use history_by_block_hash to get post-state of block (N-1). But when executing that block with the created db from the given state, I get a nonce too low error. (state is ahead of block N's nonce). This happens for the first block in the batch, first transaction. This error seems to happen only on start-up, during the pipeline process, not when it chases single blocks normally.

How I installed exex:
.install_exex("post_exec_exex", |ctx| async move { Ok(post_exec_exex(ctx)) })

How I retrieved state and created db to execute with within ExExNotification::ChainCommitted{new}
My understanding is that:

1. It takes time for the write to finish, so I loop till I get a positive response
2. Once a positive response is returned, that state is guaranteed to be final, no other changes are done to it

  let state_provider;

  loop {
      let get_result =
          ctx.provider().history_by_block_hash(new.first().parent_hash());
      if get_result.is_ok() {
          state_provider = get_result.unwrap();
          break;
      }

      tokio::time::sleep(tokio::time::Duration::from_millis(10)).await;
  };

  let mut db = State::builder()
      .with_database(StateProviderDatabase::new(&state_provider))
      .with_bundle_update()
      .without_state_clear()
      .build();

How I go about executing it:

for block in new.blocks_iter() {
    if let Err(err) =
        replay_and_index_block(&mut db, ctx.evm_config().clone(), block.clone())
    {
        panic!("exex ChainCommitted failed to process new block {:#?} with error {:#?}", block.hash(), err);
    }
}

What my replay function looks like (with custom inspector attached):

let mut inspector = TraceCollector::default();
let evm_env = evm_config.evm_env(block.header())?;
let evm = evm_config.evm_with_env_and_inspector(db, evm_env, &mut inspector);
let block_ctx = evm_config.context_for_block(&block)?;
let mut executor = evm_config.create_executor(evm, block_ctx);

executor.set_state_hook(None);
let output = executor.execute_block(block.transactions_recovered())?;

Logs:
reth log cut.txt

I've tried:

1. Using state_by_block_hash (Maybe I needed pending? Did not work)
2. Reading about some in-memory things but I don't think it should affect my historical state?
3. Used hash instead of block number for reorg/revert safety

Please advise on

1. how I can get the guaranteed correct state to replay my blocks. Am I doing something wrong?
2. Is there a way that does not need to keep looping and waiting for the state?
3. Should I not be using history_by_block_hash?
4. Is my code replaying the block, wrong?
5. How my history_by_block_hash for block N-1 have a address with higher nonce than my in block N? When block N-1 is before block N.

For past discussions, I have tried searching for "exex", "history_by_block_hash", "block_hash", "noncetoolow", "nonce too low". Was not able to find related discussions, if I missed it, please feel free to refer me to it, thank you!

[Donavern]

Could this be a corrupted database issue? I am currently trying to re-run with a fresh snapshot. On one hand, my history_by_block_hash should give me the valid state for block N-1 to execute block N, so this part of the code cannot be incorrect. On the other hand, if my node was able to process block N ~ N + X and then notify the exex, how can the historical state stored be invalid? Still quite stuck with solving this issue.

[Donavern]

I changed the state_provider retrieval to one call to history_by_block_hash, I believe the state should definitely be present when I call for it (if I have no pruning), since it should already be a committed block. The rest remains the same.

[Donavern]

Currently I've updated the code to wait for "Finish" stage checkpoint. Which means to execute a new range of blocks N ~ N + X
I wait for Finish checkpoint to be >= block N-1, which should mean block N-1 is safe to retrieve from. So i can retrieve block N-1 and proceed. Which still seems to cause nonce too high/low errors

loop {
        let checkpoint_block_number = ctx
            .provider()
            .get_stage_checkpoint(StageId::Finish)?
            .unwrap_or_default()
            .block_number;

        if checkpoint_block_number >= new.first().parent_num_hash().number {
            break;
        }

        tokio::time::sleep(Duration::from_millis(10)).await;
 }

let state_provider =
    ctx.provider().history_by_block_hash(new.first().parent_hash())?;

[Donavern]

At this point, this is what the code looks like on reth v1.9.2, which still has the nonce too high/low error which shouldn't happen because I am waiting for the height to be safe before using.

  loop {
      let checkpoint_block_number = ctx
          .provider()
          .get_stage_checkpoint(StageId::Finish)?
          .unwrap_or_default()
          .block_number;

      if checkpoint_block_number >= new.first().parent_num_hash().number {
          break;
      }

      tokio::time::sleep(Duration::from_millis(10)).await;
  }

  let state_provider =
      ctx.provider().history_by_block_hash(new.first().parent_hash())?;

  let mut db = State::builder()
      .with_database(StateProviderDatabase::new(&state_provider))
      .with_bundle_update()
      .without_state_clear()
      .build();

  for (index, block) in new.blocks_iter().enumerate() {
      if let Err(err) =
          replay_and_index_block(&mut db, ctx.evm_config().clone(), block.clone())
      {
          error!(target: "reth::cli", "exex ChainCommitted failed to process new block {:#?} with error {:#?}", block.hash(), err);
      }
  }

  ctx.events.send(ExExEvent::FinishedHeight(new.tip().num_hash()))?;

[Donavern]

This is what the logs look like, it's possible to see where the error occurs when looking through searches of "ok exex". Or directly jumping to the error with "failed to process" when replaying block 23825328 with state from 23825327 which should be safe. It can be seen that the first batch of blocks execute correctly, but the next batch of blocks fail with very different nonce values.

log.txt

[Donavern]

Instead of replaying blocks in exex, another approach was used. Directly hooking into payload_validator's execute_block function and executionstage's execute function to directly gather the data needed from the block execution during live and pipeline sync.

[Donavern]

Should wait on ctx.network().is_initially_syncing() to be done before calling history_by_block_hash (Specifically for first start-up's pipeline sync)