issues decoding deneb payloads

[alextes]

Describe the bug
we're not entirely sure of the cause, only of the obvious error / panic

{
  "err": "BUG: count '130' higher than limit '8'",
  "level": "error",
  "method": "POST",
  "msg": "http request panic: POST /eth/v1/builder/blinded_blocks",
  "service": "relay/api",
  "time": "2025-05-12T21:00:25.321447032Z",
  "trace": "goroutine 1101784 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:26 +0x5e\ngithub.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).getRouter.LoggingMiddlewareLogrus.func2.1()\n\tgithub.com/flashbots/go-utils@v0.11.0/httplogger/httplogger.go:132 +0x19e\npanic({0xfc1620?, 0xc01b844f20?})\n\truntime/panic.go:792 +0x132\ngithub.com/ferranbt/fastssz.(*Hasher).merkleizeImpl(0x1040?, {0xc01f0b7420?, 0x47dade?, 0x1300?}, {0xc01f0b7420?, 0xc01b831100?, 0x100?}, 0x8)\n\tgithub.com/ferranbt/fastssz@v0.1.4/hasher.go:364 +0x59a\ngithub.com/ferranbt/fastssz.(*Hasher).MerkleizeWithMixin(0xc000272680, 0x120, 0x81fa, 0xc01f0ad300?)\n\tgithub.com/ferranbt/fastssz@v0.1.4/hasher.go:288 +0x177\ngithub.com/ferranbt/fastssz.(*Hasher).PutBitlist(0xc000272680, {0xc01f0ad300?, 0x100?, 0x120?}, 0x800)\n\tgithub.com/ferranbt/fastssz@v0.1.4/hasher.go:232 +0x219\ngithub.com/attestantio/go-eth2-client/spec/phase0.(*Attestation).HashTreeRootWith(0xc0002e6200, {0x135b2f0, 0xc000272680})\n\tgithub.com/attestantio/go-eth2-client@v0.25.0/spec/phase0/attestation_ssz.go:114 +0x63\ngithub.com/attestantio/go-eth2-client/api/v1/deneb.(*BlindedBeaconBlockBody).HashTreeRootWith(0xc01b824780, {0x135b2f0, 0xc000272680})\n\tgithub.com/attestantio/go-eth2-client@v0.25.0/api/v1/deneb/blindedbeaconblockbody_ssz.go:507 +0x369\ngithub.com/attestantio/go-eth2-client/api/v1/deneb.(*BlindedBeaconBlock).HashTreeRootWith(0xc01c5a67e0, {0x135b2f0, 0xc000272680})\n\tgithub.com/attestantio/go-eth2-client@v0.25.0/api/v1/deneb/blindedbeaconblock_ssz.go:128 +0xc6\ngithub.com/ferranbt/fastssz.HashWithDefaultHasher({0x134a028, 0xc01c5a67e0})\n\tgithub.com/ferranbt/fastssz@v0.1.4/hasher.go:47 +0x73\ngithub.com/attestantio/go-eth2-client/api/v1/deneb.(*BlindedBeaconBlock).HashTreeRoot(...)\n\tgithub.com/attestantio/go-eth2-client@v0.25.0/api/v1/deneb/blindedbeaconblock_ssz.go:108\ngithub.com/attestantio/go-eth2-client/api.(*VersionedSignedBlindedBeaconBlock).Root(0xebf4000bc40c0002?)\n\tgithub.com/attestantio/go-eth2-client@v0.25.0/api/versionedsignedblindedbeaconblock.go:166 +0x199\ngithub.com/flashbots/mev-boost-relay/services/api.verifyBlockSignature(0xc038bc3290, {0x0, 0x0, 0x0, 0x0, 0x6a, 0x95, 0xa1, 0xa9, 0x67, ...}, ...)\n\tgithub.com/flashbots/mev-boost-relay/services/api/utils.go:169 +0x45\ngithub.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).checkProposerSignature(0x286d01c2cb71a2ad?, 0xc725ea1f12e42f32?, {0xc01f09029c?, 0x9299d0933eb1fb9f?, 0x70a4a9402cc7d2e9?})\n\tgithub.com/flashbots/mev-boost-relay/services/api/service.go:1468 +0x113\ngithub.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).handleGetPayload(0xc000244008, {0x1348858, 0xc01b60bf20}, 0xc01b824500)\n\tgithub.com/flashbots/mev-boost-relay/services/api/service.go:1774 +0x27b5\nnet/http.HandlerFunc.ServeHTTP(0xc01b8243c0?, {0x1348858?, 0xc01b60bf20?}, 0x2?)\n\tnet/http/server.go:2294 +0x29\ngithub.com/gorilla/mux.(*Router).ServeHTTP(0xc000052300, {0x1348858, 0xc01b60bf20}, 0xc01b824140)\n\tgithub.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2\ngithub.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).getRouter.LoggingMiddlewareLogrus.func2({0x134a6b8, 0xc010c94a80}, 0xc01b824140)\n\tgithub.com/flashbots/go-utils@v0.11.0/httplogger/httplogger.go:139 +0xfc\nnet/http.HandlerFunc.ServeHTTP(0x0?, {0x134a6b8?, 0xc010c94a80?}, 0x4?)\n\tnet/http/server.go:2294 +0x29\ngithub.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x1348c18, 0xc011ceb420}, 0xc01b824140)\n\tgithub.com/NYTimes/gziphandler@v1.1.1/gzip.go:336 +0x275\nnet/http.HandlerFunc.ServeHTTP(0x41b405?, {0x1348c18?, 0xc011ceb420?}, 0xc011ceb401?)\n\tnet/http/server.go:2294 +0x29\nnet/http.serverHandler.ServeHTTP({0x13456b8?}, {0x1348c18?, 0xc011ceb420?}, 0x6?)\n\tnet/http/server.go:3301 +0x8e\nnet/http.(*conn).serve(0xc01bcf6990, {0x134c458, 0xc00023a390})\n\tnet/http/server.go:2102 +0x625\ncreated by net/http.(*Server).Serve in goroutine 1\n\tnet/http/server.go:3454 +0x485\n",
  "version": ""
}

what is clear from the stacktrace is, payload api is panic-ing for signed blinded block requests (the most critical one!) but only while trying to verify a signature for a deneb block. post-fork, we shouldn't be getting any of those.

it seems in verifyBlockSignature while computing signingData.HashTreeRoot() we blow up. since we're post fork this seems easy enough to solve by checking if the payload is an electra payload and otherwise rejecting the request entirely.

why the panic happens is not clear to us.

leading explanation for us is that some old clients are still sending Deneb payloads. we particularly noted some DVT clusters sent with mev-boost 1.7 user agents but this seemed a partial explanation at best.

Expected behavior

1. to reject payloads only valid pre-fork, post-fork would be nice
2. to not panic when verifying old signatures

Which version are you using:
we run a slightly modified forked version. although the above seems unlikely due to our modifications it could be. would be interesting if other relays, specifically those running this code saw the same error.

[alextes]

deployed a new version with the rejection logic, if errors disappear i'll share it here. will take a day or so to be sure.

[metachris]

That's the expanded stack trace:

  "trace": "goroutine 1101784 [running]:
 runtime/debug.Stack()
 	runtime/debug/stack.go:26 +0x5e
 github.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).getRouter.LoggingMiddlewareLogrus.func2.1()
 	github.com/flashbots/go-utils@v0.11.0/httplogger/httplogger.go:132 +0x19e
 panic({0xfc1620?, 0xc01b844f20?})
 	runtime/panic.go:792 +0x132
 github.com/ferranbt/fastssz.(*Hasher).merkleizeImpl(0x1040?, {0xc01f0b7420?, 0x47dade?, 0x1300?}, {0xc01f0b7420?, 0xc01b831100?, 0x100?}, 0x8)
 	github.com/ferranbt/fastssz@v0.1.4/hasher.go:364 +0x59a
 github.com/ferranbt/fastssz.(*Hasher).MerkleizeWithMixin(0xc000272680, 0x120, 0x81fa, 0xc01f0ad300?)
 	github.com/ferranbt/fastssz@v0.1.4/hasher.go:288 +0x177
 github.com/ferranbt/fastssz.(*Hasher).PutBitlist(0xc000272680, {0xc01f0ad300?, 0x100?, 0x120?}, 0x800)
 	github.com/ferranbt/fastssz@v0.1.4/hasher.go:232 +0x219
 github.com/attestantio/go-eth2-client/spec/phase0.(*Attestation).HashTreeRootWith(0xc0002e6200, {0x135b2f0, 0xc000272680})
 	github.com/attestantio/go-eth2-client@v0.25.0/spec/phase0/attestation_ssz.go:114 +0x63
 github.com/attestantio/go-eth2-client/api/v1/deneb.(*BlindedBeaconBlockBody).HashTreeRootWith(0xc01b824780, {0x135b2f0, 0xc000272680})
 	github.com/attestantio/go-eth2-client@v0.25.0/api/v1/deneb/blindedbeaconblockbody_ssz.go:507 +0x369
 github.com/attestantio/go-eth2-client/api/v1/deneb.(*BlindedBeaconBlock).HashTreeRootWith(0xc01c5a67e0, {0x135b2f0, 0xc000272680})
 	github.com/attestantio/go-eth2-client@v0.25.0/api/v1/deneb/blindedbeaconblock_ssz.go:128 +0xc6
 github.com/ferranbt/fastssz.HashWithDefaultHasher({0x134a028, 0xc01c5a67e0})
 	github.com/ferranbt/fastssz@v0.1.4/hasher.go:47 +0x73
 github.com/attestantio/go-eth2-client/api/v1/deneb.(*BlindedBeaconBlock).HashTreeRoot(...)
 	github.com/attestantio/go-eth2-client@v0.25.0/api/v1/deneb/blindedbeaconblock_ssz.go:108
 github.com/attestantio/go-eth2-client/api.(*VersionedSignedBlindedBeaconBlock).Root(0xebf4000bc40c0002?)
 	github.com/attestantio/go-eth2-client@v0.25.0/api/versionedsignedblindedbeaconblock.go:166 +0x199
 github.com/flashbots/mev-boost-relay/services/api.verifyBlockSignature(0xc038bc3290, {0x0, 0x0, 0x0, 0x0, 0x6a, 0x95, 0xa1, 0xa9, 0x67, ...}, ...)
 	github.com/flashbots/mev-boost-relay/services/api/utils.go:169 +0x45
 github.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).checkProposerSignature(0x286d01c2cb71a2ad?, 0xc725ea1f12e42f32?, {0xc01f09029c?, 0x9299d0933eb1fb9f?, 0x70a4a9402cc7d2e9?})
 	github.com/flashbots/mev-boost-relay/services/api/service.go:1468 +0x113
 github.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).handleGetPayload(0xc000244008, {0x1348858, 0xc01b60bf20}, 0xc01b824500)
 	github.com/flashbots/mev-boost-relay/services/api/service.go:1774 +0x27b5
 net/http.HandlerFunc.ServeHTTP(0xc01b8243c0?, {0x1348858?, 0xc01b60bf20?}, 0x2?)
 	net/http/server.go:2294 +0x29
 github.com/gorilla/mux.(*Router).ServeHTTP(0xc000052300, {0x1348858, 0xc01b60bf20}, 0xc01b824140)
 	github.com/gorilla/mux@v1.8.1/mux.go:212 +0x1e2
 github.com/flashbots/mev-boost-relay/services/api.(*RelayAPI).getRouter.LoggingMiddlewareLogrus.func2({0x134a6b8, 0xc010c94a80}, 0xc01b824140)
 	github.com/flashbots/go-utils@v0.11.0/httplogger/httplogger.go:139 +0xfc
 net/http.HandlerFunc.ServeHTTP(0x0?, {0x134a6b8?, 0xc010c94a80?}, 0x4?)
 	net/http/server.go:2294 +0x29
 github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x1348c18, 0xc011ceb420}, 0xc01b824140)
 	github.com/NYTimes/gziphandler@v1.1.1/gzip.go:336 +0x275
 net/http.HandlerFunc.ServeHTTP(0x41b405?, {0x1348c18?, 0xc011ceb420?}, 0xc011ceb401?)
 	net/http/server.go:2294 +0x29
 net/http.serverHandler.ServeHTTP({0x13456b8?}, {0x1348c18?, 0xc011ceb420?}, 0x6?)
 	net/http/server.go:3301 +0x8e
 net/http.(*conn).serve(0xc01bcf6990, {0x134c458, 0xc00023a390})
 	net/http/server.go:2102 +0x625
 created by net/http.(*Server).Serve in goroutine 1
 	net/http/server.go:3454 +0x485
 ",

[metachris]

mev-boost-relay should reject builder submissions with a non-electra payload here:

mev-boost-relay/services/api/service.go

Line 1861 in f62940b

if api.isElectra(submission.BidTrace.Slot) && payload.Electra == nil {

(although it's not immediately obvious that this check is not error-free -- i.e. could a Deneb payload pass this check somehow)?

If that checks works as expected, it seems mev-boost-relay wouldn't accept or deliver a non-Pectra bid/payload. Older mev-boost versions do multiplex getPayload to all relays, and it might be the case that some relay let's those blocks through, and some old mev-boost versions trying to submit.

It would be helpful to know example slot(s) where this happend.

[metachris]

I would guess that there is an electra block being sent but it is picked up as a deneb block for some reason (a guess by a JSON decoder? An incorrect version field or Eth-Consensus-Value header?). Only a guess, mind.

[metachris]

Spot checked some of the requests that error out, and all of them use old non-pectra mev-boost versions!

timestamp	slot	content encoding	mev-boost version	notes
2025-05-14T05:23:27Z	11698015	JSON	1.7.1	not missed
2025-05-14T00:46:49Z	11696632	JSON	1.8.1	not missed
2025-05-13T23:05:37Z	11696126	JSON	1.8	not missed
2025-05-13T13:43:54Z	11693312	JSON	1.8.1	not missed
2025-05-13T13:19:28Z	11693195	JSON	1.7.1	not missed
2025-05-13T11:59:49Z	11692797	JSON	-	not missed
2025-05-13T10:10:01Z	11692248	JSON	1.7	not missed
2025-05-13T09:25:49Z	11692027	JSON	1.7.1	not missed
2025-05-13T01:47:45Z	11689736	JSON	1.8.1	not missed
2025-05-13T00:08:37Z	11689241	JSON	1.7.1	not missed

[metachris]

Code trail for deserialization:

• getPayload:

  mev-boost-relay/services/api/service.go

  Line 1418 in f62940b

  err = payload.Unmarshal(body, proposerContentType, proposerEthConsensusVersion)

• Unmarshalling switch, based on content type JSON:

  mev-boost-relay/common/types_spec.go

  Lines 622 to 624 in f62940b

  	case "application/json":
  	return json.NewDecoder(bytes.NewReader(input)).Decode(r)
  	}

• JSON Unmarshalling of the type:

  mev-boost-relay/common/types_spec.go

  Lines 572 to 594 in f62940b

  	func (r *VersionedSignedBlindedBeaconBlock) UnmarshalJSON(input []byte) error {
  	var err error
  	electraBlock := new(eth2ApiV1Electra.SignedBlindedBeaconBlock)
  	if err = json.Unmarshal(input, electraBlock); err == nil {
  	r.Version = spec.DataVersionElectra
  	r.Electra = electraBlock
  	return nil
  	}
  	denebBlock := new(eth2ApiV1Deneb.SignedBlindedBeaconBlock)
  	if err = json.Unmarshal(input, denebBlock); err == nil {
  	r.Version = spec.DataVersionDeneb
  	r.Deneb = denebBlock
  	return nil
  	}
  	capellaBlock := new(eth2ApiV1Capella.SignedBlindedBeaconBlock)
  	if err = json.Unmarshal(input, capellaBlock); err == nil {
  	r.Version = spec.DataVersionCapella
  	r.Capella = capellaBlock
  	return nil
  	}
  	return errors.Wrap(err, "failed to unmarshal SignedBlindedBeaconBlock")
  	}

  • Specifically, the Electra deserialization fails here:

    mev-boost-relay/common/types_spec.go

    Lines 574 to 579 in f62940b

    	electraBlock := new(eth2ApiV1Electra.SignedBlindedBeaconBlock)
    	if err = json.Unmarshal(input, electraBlock); err == nil {
    	r.Version = spec.DataVersionElectra
    	r.Electra = electraBlock
    	return nil
    	}

  • Making it deserialize into a Deneb block
• The go-eth2-client code for deserialization into Electra payload is here: https://github.com/attestantio/go-eth2-client/blob/v0.25.0/api/v1/electra/signedblindedbeaconblock_json.go#L41-L48

On a quick glance, I’m not sure why it fails to deserialize into a Electra block 🤔

For reference, this is the serialization code for mev-boost v1.7.1:

• https://github.com/flashbots/mev-boost/blob/v1.7.1/server/service.go#L807-L808
• https://github.com/attestantio/go-eth2-client/blob/v0.19.10/api/v1/deneb/signedblindedbeaconblock_json.go#L33-L38