libp2p being clogged due to some connectivy issue

[clangenb]

I am not sure if #9094 is 100% resolved yet. According to these logs, I can see a clogging issue around the session change.

However, there seem to have been a few networking issues before, although our monitoring did not record any issues, and the provider confirmed that there was no downtime. I have attached the relevant logs.

Explore-logs-2025-10-30 08_34_39.txt

[sandreim]

I think this is a different issue. #9094 deals with bitfield gossip messages. But in your case I see quite. a few: 2025-10-29 23:49:55 /91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3/req_chunk/2: failed to send request to PeerId("12D3KooWRvEMACVHQRjU35Ufb8yrfxdVwK5ZHvWgmYCUkJYknKZ4"): ChannelClogged . These are not gossip, but requests for a specific chunk coming from availability recovery or availability distribution.

These could be because of some network connectivity issue ? The messages appear after a few blocks into a new session.

CC @alexggh

[alexggh]

I think this is a different issue. #9094

This is completely a different issue.

it seems like the rate of sending req_chunks is higher than what the node can send, warning is coming from litep2p, fyi @paritytech/networking.

Maybe this const DEFAULT_CHANNEL_SIZE: usize = 4096usize; is probably too small to handle burst and you get those warnings.

[clangenb]

I can agree that networking issues might be the origin of that. However, we also run other nodes in remote locations, and obviously the networking is not stable all the time, but we never saw that behavior that it couldn't recover so far - but two times on v1.20.1.

Edit: Updated the issue name to reflect libp2p domain

[clangenb]

We had another incident on the same machine the same day. I have quite extensively checked the logs, and I could not see anything else than potential connectivity issues. However, I do fail to understand a few things, which I hope l can get answers for:

The nodes were continuously connected to our monitoring and also visible on the public telemetry. So it can't be a general connectivity issue, and we do have quite some peers that we were still being connected to, e.g. log lines:

1761815667368	2025-10-30 09:14:27 💤 Idle (10 peers), best: #28420247 (0x7e90…3b8b), finalized #28420244 (0xf1b4…cb6a), ⬇ 1.2MiB/s ⬆ 1.5MiB/s    
1761815667718	2025-10-30 09:14:27 Some network error occurred when fetching erasure chunk origin=Public(dcefd68753bc37badbd0ca393ea19cf77d4de23d1ca95ecd4e9ca2715626460f (15zgnKeD...)) relay_parent=0xf1b4044d1587270f3e9aea69c9d4de74268a6528cbb52ddb44c053dbdf20cb6a group_index=GroupIndex(115) session_index=11896 chunk_index=ValidatorIndex(325) candidate_hash=0x1465289fe20a69bd9f0611e0e2569de0fb84ad732e4e1255c50f81e4c5f83922 err=Refused traceID=27109805862028521707598685625256418784
1761815669820	2025-10-30 09:14:29 Some network error occurred when fetching erasure chunk origin=Public(f62975ed7059b969dd5ec94075a6649b95b84547014016e3e2db185a9256cd2a (16Zm6a2V...)) relay_parent=0x79516b88e850bec287f4732a2da241c0677d3f01a6b3ce3c218b5bf99a17dfac group_index=GroupIndex(24) session_index=11896 chunk_index=ValidatorIndex(325) candidate_hash=0x4b86c4a4f571dd57109d38f90679f2780676154b554567edcbfd7233d3675619 err=Refused traceID=100391855884574806709665694288394711672

However, we see that we cannot connect to our group peers here:

1761818108734	2025-10-30 09:55:08 Cluster has too many pending statements, something wrong with our connection to our group peers
1761818108734					Restart might be needed if validator gets 0 backing rewards for more than 3-4 consecutive sessions pending_statements={ValidatorIndex(327): {(ValidatorIndex(325), CompactStatement::Seconded(0x4a14f388e2170cc270b28badba4c2fd2c805f39c2e2b29d28945e76c9666ce04))}, ValidatorIndex(329): {(ValidatorIndex(325), CompactStatement::Seconded(0x4a14f388e2170cc270b28badba4c2fd2c805f39c2e2b29d28945e76c9666ce04))}, ValidatorIndex(326): {(ValidatorIndex(325), CompactStatement::Seconded(0x4a14f388e2170cc270b28badba4c2fd2c805f39c2e2b29d28945e76c9666ce04))}, ValidatorIndex(328): {(ValidatorIndex(325), CompactStatement::Seconded(0x4a14f388e2170cc270b28badba4c2fd2c805f39c2e2b29d28945e76c9666ce04))}, ValidatorIndex(325): {(ValidatorIndex(325), CompactStatement::Seconded(0x4a14f388e2170cc270b28badba4c2fd2c805f39c2e2b29d28945e76c9666ce04))}} parent_hash=0x5d8ec781591b3f38d04cd8c20e51981b7926115f1523914ab6ea0497e802ba66
1761818109084	2025-10-30 09:55:09 maintain txs=(2, 0) a=1 i=4 views=[(28420650, 0, 0)] event=Finalized { hash: 0x18d34f00c2c94f9ed9f894812497098e08b006469c849389fc51b8208ca737cf, tree_route: [] } duration=194.322µs

And yet another error, what might lead to the Refused error?

1761818209298	2025-10-30 09:56:49 fetch_pov_job err=FetchPoV(NetworkError(Refused)) para_id=Id(3421) pov_hash=0x1556d659b3614266d92b83405a9f9a077f6a45822e9e3cda73f3d06e4b3d6f88 authority_id=Public(8a52ea9f343152113b71b8328a88787bcdeb2fd5d18f5dba31264017cfd74d69 (148NEtT6...))

In our history in running validators, we only had this kind of error, when our monitoring detected networking issues too, it is a first time for us that our connectivity checks work, but the P2P networking is having issues, we have migrated away from this machine as we do not understand where this error is coming from.

Explore-logs-2025-11-01 12_05_07.txt

[dmitry-markin]

Refused might mean that the remote node's listening port is closed or local connectivity issues, but also that the remote peer has banned the local node for some misbehavior. I would suspect the latter if this is a recurring issue. The most reliable way to tell what happened is to check the banning node logs, so if you also control some of the nodes refusing the local node that would greatly simplify the debugging.