bug: race condition b/w flashblocks vs. fallback `getPayload` causes the sequencer to stall

[0x416e746f6e]

tl;dr: rollup-boost can return the same block for two different payload ids, which causes the sequencer to log "Sequencer paused until new events" and stall forever (only restart would revive it).

the scenario is most likely to occur during the periods when the sequencer is incrementally advancing safe head upon reading batcher's channel (see: ethereum-optimism/optimism#15393)

what can happen is:

• N: when handling getPayload from the sequencer rollup-boost, for whatever reason, instead of returning the block that it had assembled from the flashblocks stream, falls back to the engine api, and issues the getPayload call to the builder
  • builder returns a valid block, and that's what is handed over to the sequencer
• N+1: then for the next getPayload from the sequencer rollup-boost would send still the same block N, but this time it's the one it assembled from the flashblocks stream
  • sequencer receives a block with the hash it already knows, and the logic here is not triggered, which blocks its event loop forever

example

jrpc calls on rollup-boost side:

note that for 2 consecutive payload ids 0x03af34acbbd4615d and 0x03464ebe9ab36481 the same block 0x07ae1e557c92eb007c493ff13775b3ab950a328eefdb0e6de7fc68bbd1196278 is returned by rollup-boost

jrpc calls on builder side:

the response for payload id 0x03af34acbbd4615d actually came from the builder

[0x416e746f6e]

could be related to #149

[SozinM]

Should be closed with #307, will reopen if it not the case