[BUG] Multiple nodes with same IP and PORT

[esebole]

Describe the bug

We have a test case in which 2 out 3 primaries are deleted in the same time - short gap between deletion in reality. In majority of cases this gap is enough for cluster to recover (time for failovers). Anyway in failure runs (1/50) I am observing a strange behavior in which multiple nodes have same IP and PORT: 127.0.0.1:6380. It doesn't seam right. Have a look at the logs below which I have.

I1010 23:52:09.100471  18:M Address updated for node dad564b61f8758c0d29328b5803edcdaa0811c6f (), now 127.0.0.1:6380
I1010 23:52:09.100939  d933a2d1b785fa934f1a5efae9243a937b522ea1 127.0.0.1:6380@16380 myself,master - 0 0 9 connected 10923-16383
I1010 23:52:09.100947  756d2c3a79c1dfba5df86ccff020e0a72475a47d 192.168.37.165:6380@16380 master - 0 1760140328554 7 connected 5462-10922
I1010 23:52:09.100950  7fcfe27413cbc06f71897ccd5f9ff4b1083cfbbe 192.168.51.41:6380@16380 slave 756d2c3a79c1dfba5df86ccff020e0a72475a47d 0 1760140328554 7 connected
I1010 23:52:09.100926  dad564b61f8758c0d29328b5803edcdaa0811c6f 127.0.0.1:6380@16380 slave a6126962691ab37568773919497b6498254f26fd 1760140328152 1760140328000 13 disconnected
I1010 23:52:09.100943  a6126962691ab37568773919497b6498254f26fd 192.168.132.112:6380@16380 master - 0 1760140328857 13 connected 0-5461

To reproduce

N/A. I will try to reproduce the issue with lower logging level so the GOSSIPs will be visible, but maybe what I already shared gives any clue if this can be improved.

Expected behavior

I think there should be no node in the cluster with same IP and PORT.

Additional information

Full log from affected pod:

I1010 23:40:39.716347  18:S Start of election delayed for 0 milliseconds (rank #0, offset 650332392).
I1010 23:40:39.716473  18:S Starting a failover election for epoch 9, node config epoch is 8
I1010 23:40:39.716279  18:S Forced failover user request accepted (user request from 'id=141 addr=192.168.5.54:37738 laddr=192.168.5.52:6380 fd=44 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=20 multi-mem=0 rbs=16384 rbp=16384 obl=0
 oll=0 omem=0 tot-mem=17324 events=r cmd=cluster|failover user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=232 tot-net-out=2971 tot-cmds=5').
I1010 23:40:39.720222  18:M Discarding previously cached primary state.
I1010 23:40:39.720208  18:S configEpoch set to 9 after successful failover
I1010 23:40:39.720185  18:S Failover election won: I'm the new primary.
I1010 23:40:39.720214  18:M Connection with primary lost.
I1010 23:40:39.720218  18:M Caching the disconnected primary state.
I1010 23:40:39.720228  18:M Setting secondary replication ID to fc2c29faebd3533e0f3b15052d651cb656853440, valid up to offset: 650332393. New replication ID is aa2dd4f2e298e5a3122a1f1646b7de61218adb0b
I1010 23:40:39.723631 
I1010 23:40:39.723589  dad564b61f8758c0d29328b5803edcdaa0811c6f 192.168.51.35:6380@16380 slave 6f038b2fa88fe9c6bbcd9981db7c7ea458b42d44 0 1760139639092 1 connected
I1010 23:40:39.723618  d933a2d1b785fa934f1a5efae9243a937b522ea1 192.168.5.52:6380@16380 myself,master - 0 0 9 connected 10923-16383
I1010 23:40:39.723625  756d2c3a79c1dfba5df86ccff020e0a72475a47d 192.168.37.165:6380@16380 master - 0 1760139639596 7 connected 5462-10922
I1010 23:40:39.723628  6f038b2fa88fe9c6bbcd9981db7c7ea458b42d44 192.168.132.116:6380@16380 master - 0 1760139639193 1 connected 0-5461
I1010 23:40:39.723613  256f23c60e389a6edfd5a657ca2c143e50c42830 192.168.51.38:6380@16380 master - 0 1760139639596 8 connected
I1010 23:40:39.723621  a6126962691ab37568773919497b6498254f26fd 192.168.132.112:6380@16380 slave 756d2c3a79c1dfba5df86ccff020e0a72475a47d 0 1760139639193 7 connected
I1010 23:40:39.732682  18:M Partial resynchronization not accepted: Requested offset for second ID was 650332995, but I can reply up to 650332393
I1010 23:40:39.732651  18:M Replica 192.168.51.38:6380 asks for synchronization
I1010 23:40:39.732691  18:M Replica 192.168.51.38:6380 is capable of dual channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated RDB channel.
I1010 23:40:39.740857  18:M Starting BGSAVE for SYNC with target: replicas sockets using: dual-channel
I1010 23:40:39.740838  18:M Replica 192.168.51.38:6380 asks for synchronization
I1010 23:40:39.740863  18:M Sending to replica 192.168.51.38:6380 RDB end offset 650332392 and client-id 147
I1010 23:40:39.745611  18:M Background RDB transfer started by pid 32 to direct socket to replica
I1010 23:40:39.748410  18:M Replica 192.168.51.38:6380 asks for synchronization
I1010 23:40:39.748450  18:M Partial resynchronization request from 192.168.51.38:6380 accepted. Sending 0 bytes of backlog starting from offset 650332393.
I1010 23:40:39.764787  18:M Connection with replica client id #143 lost.
I1010 23:40:39.765286  18:M Replica 192.168.51.38:6380 rdb channel disconnected.
W1010 23:40:39.797651  18:M Background RDB transfer error
I1010 23:40:42.176969  18:M Successfully completed handshake with cba6f96b9c5d2cf97ff2112a77967a0c982873be ()
W1010 23:40:42.176934  18:M Failover election in progress for epoch 9, but received a claim from node cba6f96b9c5d2cf97ff2112a77967a0c982873be () with an equal or higher epoch 10. Resetting the election since we cannot win an election in the past.
I1010 23:40:42.177122  18:M Address updated for node cba6f96b9c5d2cf97ff2112a77967a0c982873be (), now 192.168.37.138:6380
I1010 23:40:42.180548  756d2c3a79c1dfba5df86ccff020e0a72475a47d 192.168.37.165:6380@16380 master - 0 1760139641648 7 connected 5462-10922
I1010 23:40:42.180503  cba6f96b9c5d2cf97ff2112a77967a0c982873be 192.168.37.138:6380@16380 master - 0 1760139642157 10 disconnected
I1010 23:40:42.180530  dad564b61f8758c0d29328b5803edcdaa0811c6f 192.168.51.35:6380@16380 slave 6f038b2fa88fe9c6bbcd9981db7c7ea458b42d44 0 1760139641648 1 connected
I1010 23:40:42.180544  a6126962691ab37568773919497b6498254f26fd 192.168.132.112:6380@16380 slave 756d2c3a79c1dfba5df86ccff020e0a72475a47d 0 1760139641211 7 connected
I1010 23:40:42.180539  d933a2d1b785fa934f1a5efae9243a937b522ea1 192.168.5.52:6380@16380 myself,master - 0 0 9 connected 10923-16383
I1010 23:40:42.180551  6f038b2fa88fe9c6bbcd9981db7c7ea458b42d44 192.168.132.116:6380@16380 master - 0 1760139642000 1 connected 0-5461
I1010 23:40:42.180554 
I1010 23:40:42.255927  18:M Node cba6f96b9c5d2cf97ff2112a77967a0c982873be () is no longer primary of shard b97108364721ecad6dfcc1932290c9ba2e321bca; removed all 0 slot(s) it used to own
I1010 23:40:42.255958  18:M Node cba6f96b9c5d2cf97ff2112a77967a0c982873be () is now part of shard b24ffc0709aee18910992a47668946085bc63936
I1010 23:40:42.255964  18:M Node cba6f96b9c5d2cf97ff2112a77967a0c982873be () is now a replica of node d933a2d1b785fa934f1a5efae9243a937b522ea1 () in shard b24ffc0709aee18910992a47668946085bc63936
I1010 23:40:42.259767  18:M Replica 192.168.37.138:6380 is capable of dual channel synchronization, and partial sync isn't possible. Full sync will continue with dedicated RDB channel.
I1010 23:40:42.259761  18:M Full resync requested by replica 192.168.37.138:6380
I1010 23:40:42.259749  18:M Replica 192.168.37.138:6380 asks for synchronization
I1010 23:40:42.263598  18:M Replica 192.168.37.138:6380 asks for synchronization
I1010 23:40:42.263622  18:M Starting BGSAVE for SYNC with target: replicas sockets using: dual-channel
I1010 23:40:42.263627  18:M Sending to replica 192.168.37.138:6380 RDB end offset 650348316 and client-id 186
I1010 23:40:42.267637  18:M Background RDB transfer started by pid 33 to direct socket to replica
I1010 23:40:42.269417  18:M Partial resynchronization request from 192.168.37.138:6380 accepted. Sending 0 bytes of backlog starting from offset 650348317.
I1010 23:40:42.269370  18:M Replica 192.168.37.138:6380 asks for synchronization
I1010 23:40:42.344211  log catcher is delaying logging cluster state
I1010 23:40:42.343905  18:M Replica 192.168.37.138:6380 rdb channel disconnected.
I1010 23:40:42.344175  18:M Synchronization with replica 192.168.37.138:6380 succeeded
I1010 23:40:42.345081  33:C Fork CoW for RDB: current 3 MB, peak 3 MB, average 2 MB
I1010 23:40:42.352481  18:M Background RDB transfer terminated with success
I1010 23:40:43.345034  dad564b61f8758c0d29328b5803edcdaa0811c6f 192.168.51.35:6380@16380 slave 6f038b2fa88fe9c6bbcd9981db7c7ea458b42d44 0 1760139643058 1 connected
I1010 23:40:43.345039  d933a2d1b785fa934f1a5efae9243a937b522ea1 192.168.5.52:6380@16380 myself,master - 0 0 9 connected 10923-16383
I1010 23:40:43.345044  a6126962691ab37568773919497b6498254f26fd 192.168.132.112:6380@16380 slave 756d2c3a79c1dfba5df86ccff020e0a72475a47d 0 1760139643058 7 connected
I1010 23:40:43.345051  6f038b2fa88fe9c6bbcd9981db7c7ea458b42d44 192.168.132.116:6380@16380 master - 0 1760139643000 1 connected 0-5461
I1010 23:40:43.345054 
I1010 23:40:43.345001  cba6f96b9c5d2cf97ff2112a77967a0c982873be 192.168.37.138:6380@16380 slave d933a2d1b785fa934f1a5efae9243a937b522ea1 0 1760139643260 9 connected
I1010 23:40:43.345048  756d2c3a79c1dfba5df86ccff020e0a72475a47d 192.168.37.165:6380@16380 master - 0 1760139643058 7 connected 5462-10922
I1010 23:42:27.966860  18:M Failover auth granted to dad564b61f8758c0d29328b5803edcdaa0811c6f () for epoch 11
I1010 23:42:32.145391  18:M Node 7fcfe27413cbc06f71897ccd5f9ff4b1083cfbbe () is now a replica of node dad564b61f8758c0d29328b5803edcdaa0811c6f () in shard bec51287c31b192c3a4a1936493523ca0882f87d
I1010 23:42:32.145377  18:M Node 7fcfe27413cbc06f71897ccd5f9ff4b1083cfbbe () is no longer primary of shard 14d7218516be3bff8259ee30bc3b6b57fa835673; removed all 0 slot(s) it used to own
I1010 23:42:32.145385  18:M Node 7fcfe27413cbc06f71897ccd5f9ff4b1083cfbbe () is now part of shard bec51287c31b192c3a4a1936493523ca0882f87d
I1010 23:42:32.145339  18:M Successfully completed handshake with 7fcfe27413cbc06f71897ccd5f9ff4b1083cfbbe ()
I1010 23:42:32.438524  d933a2d1b785fa934f1a5efae9243a937b522ea1 192.168.5.52:6380@16380 myself,master - 0 0 9 connected 10923-16383
I1010 23:42:32.438530  a6126962691ab37568773919497b6498254f26fd 192.168.132.112:6380@16380 slave 756d2c3a79c1dfba5df86ccff020e0a72475a47d 0 1760139752129 7 connected
I1010 23:42:32.438461  cba6f96b9c5d2cf97ff2112a77967a0c982873be 192.168.37.138:6380@16380 slave d933a2d1b785fa934f1a5efae9243a937b522ea1 0 1760139752000 9 connected
I1010 23:42:32.438535  756d2c3a79c1dfba5df86ccff020e0a72475a47d 192.168.37.165:6380@16380 master - 0 1760139752028 7 connected 5462-10922
I1010 23:42:32.438540  7fcfe27413cbc06f71897ccd5f9ff4b1083cfbbe 192.168.51.41:6380@16380 slave dad564b61f8758c0d29328b5803edcdaa0811c6f 0 1760139752432 11 disconnected
I1010 23:42:32.437782  18:M Address updated for node 7fcfe27413cbc06f71897ccd5f9ff4b1083cfbbe (), now 192.168.51.41:6380
I1010 23:42:32.438544 
I1010 23:42:32.438490  dad564b61f8758c0d29328b5803edcdaa0811c6f 192.168.51.35:6380@16380 master - 0 1760139752331 11 connected 0-5461
I1010 23:42:34.403031  18:M Node 7fcfe27413cbc06f71897ccd5f9ff4b1083cfbbe () is now a replica of node 756d2c3a79c1dfba5df86ccff020e0a72475a47d () in shard 2f93c933f455fdff696bd292b7d9b7330ee83314
I1010 23:42:34.404257  18:M Node a6126962691ab37568773919497b6498254f26fd () is now a replica of node dad564b61f8758c0d29328b5803edcdaa0811c6f () in shard bec51287c31b192c3a4a1936493523ca0882f87d
I1010 23:52:07.940500       1 signal.go:30] Signal received: terminated
I1010 23:52:07.940665       1 main.go:38] Shutdown handler called
E1010 23:52:07.940792       1 redisnode.go:325] http server stop error: http: Server closed
I1010 23:52:07.975305       1 admin.go:321] Failover by address: 192.168.5.52:6380
I1010 23:52:07.984725       1 admin.go:345] Starting failover for node d933a2d1b785fa934f1a5efae9243a937b522ea1
W1010 23:52:08.019923       1 clusterstatus.go:223] inconsistent cluster view
I1010 23:52:08.019964       1 clusterstatus.go:224] consolidated signature:
I1010 23:52:08.021038       1 clusterstatus.go:231] node 756d signature:
I1010 23:52:08.021026       1 clusterstatus.go:227]
I1010 23:52:08.020986       1 clusterstatus.go:227] 756d [5462-10922]
I1010 23:52:08.023412       1 clusterstatus.go:234] d933 [10923-16383]
I1010 23:52:08.023416       1 clusterstatus.go:234]
I1010 23:52:08.023426       1 clusterstatus.go:231] node 7fcf signature:
I1010 23:52:08.023407       1 clusterstatus.go:234] dad5 [0-5461]
I1010 23:52:08.023376       1 clusterstatus.go:234] 756d [5462-10922]
I1010 23:52:08.029074       1 clusterstatus.go:234] 756d [5462-10922]
I1010 23:52:08.029084       1 clusterstatus.go:234] d933 [10923-16383]
I1010 23:52:08.029031       1 clusterstatus.go:234] dad5 [0-5461]
I1010 23:52:08.029091       1 clusterstatus.go:234]
I1010 23:52:08.029103       1 clusterstatus.go:231] node a612 signature:
I1010 23:52:08.031447       1 clusterstatus.go:234]
I1010 23:52:08.031411       1 clusterstatus.go:234] 756d [5462-10922]
I1010 23:52:08.031439       1 clusterstatus.go:234] d933 [10923-16383]
I1010 23:52:08.031444       1 clusterstatus.go:234] dad5 [0-5461]
I1010 23:52:08.031454       1 clusterstatus.go:231] node cba6 signature:
I1010 23:52:08.034195       1 admin.go:351] Found 1 slave nodes
I1010 23:52:08.034182       1 clusterstatus.go:234] d933 [10923-16383]
I1010 23:52:08.034186       1 clusterstatus.go:234] 756d [5462-10922]
I1010 23:52:08.034202       1 admin.go:367] Attempting failover for slave node cba6f96b9c5d2cf97ff2112a77967a0c982873be
I1010 23:52:08.034157       1 clusterstatus.go:234] dad5 [0-5461]
I1010 23:52:08.034189       1 clusterstatus.go:234]
I1010 23:52:08.039138  18:M Failover auth granted to a6126962691ab37568773919497b6498254f26fd () for epoch 13
I1010 23:52:08.051635       1 admin.go:389] Failover command successfully sent to cba6f96b9c5d2cf97ff2112a77967a0c982873be
I1010 23:52:08.051768  18:M Manual failover requested by replica cba6f96b9c5d2cf97ff2112a77967a0c982873be ().
I1010 23:52:08.051678       1 admin.go:397] Starting to wait for slot transfer from master=(d933a2d1b785fa934f1a5efae9243a937b522ea1, 192.168.5.52:6380) to replica=(cba6f96b9c5d2cf97ff2112a77967a0c982873be, 192.168.37.138:6380) to complete, total slots to transfer: 5461
W1010 23:52:08.059995  18:M Failover auth denied to cba6f96b9c5d2cf97ff2112a77967a0c982873be : can't vote about this primary before 3997 milliseconds
I1010 23:52:08.059971  18:M Failover auth granted to cba6f96b9c5d2cf97ff2112a77967a0c982873be () for epoch 14
I1010 23:52:08.060182       1 admin.go:417] Waiting for failover to complete slot transfer...
I1010 23:52:08.091436  18:M Node dad564b61f8758c0d29328b5803edcdaa0811c6f () is now a replica of node a6126962691ab37568773919497b6498254f26fd () in shard bec51287c31b192c3a4a1936493523ca0882f87d
I1010 23:52:08.091404  18:M Address updated for node dad564b61f8758c0d29328b5803edcdaa0811c6f (), now 127.0.0.1:6380
I1010 23:52:08.096713  d933a2d1b785fa934f1a5efae9243a937b522ea1 127.0.0.1:6380@16380 myself,master - 0 0 9 connected 10923-16383
I1010 23:52:08.096732 
I1010 23:52:08.096708  dad564b61f8758c0d29328b5803edcdaa0811c6f 127.0.0.1:6380@16380 slave a6126962691ab37568773919497b6498254f26fd 0 1760140327847 13 disconnected
I1010 23:52:08.096717  a6126962691ab37568773919497b6498254f26fd 192.168.132.112:6380@16380 master - 0 1760140327140 13 connected 0-5461
I1010 23:52:08.096723  7fcfe27413cbc06f71897ccd5f9ff4b1083cfbbe 192.168.51.41:6380@16380 slave 756d2c3a79c1dfba5df86ccff020e0a72475a47d 0 1760140327544 7 connected
I1010 23:52:08.096720  756d2c3a79c1dfba5df86ccff020e0a72475a47d 192.168.37.165:6380@16380 master - 0 1760140327544 7 connected 5462-10922
I1010 23:52:08.096687  cba6f96b9c5d2cf97ff2112a77967a0c982873be 192.168.37.138:6380@16380 slave d933a2d1b785fa934f1a5efae9243a937b522ea1 0 1760140328049 9 connected
I1010 23:52:08.151389  18:M Connection with replica 192.168.37.138:6380 lost.
I1010 23:52:08.154041  18:M PONG contains mismatching sender ID. About node dad564b61f8758c0d29328b5803edcdaa0811c6f () in shard bec51287c31b192c3a4a1936493523ca0882f87d added 1532191 ms ago, having flags 3074
I1010 23:52:09.100904  cba6f96b9c5d2cf97ff2112a77967a0c982873be 192.168.37.138:6380@16380 slave d933a2d1b785fa934f1a5efae9243a937b522ea1 1760140329058 1760140328152 9 connected
I1010 23:52:09.100471  18:M Address updated for node dad564b61f8758c0d29328b5803edcdaa0811c6f (), now 127.0.0.1:6380
I1010 23:52:09.100939  d933a2d1b785fa934f1a5efae9243a937b522ea1 127.0.0.1:6380@16380 myself,master - 0 0 9 connected 10923-16383
I1010 23:52:09.100947  756d2c3a79c1dfba5df86ccff020e0a72475a47d 192.168.37.165:6380@16380 master - 0 1760140328554 7 connected 5462-10922
I1010 23:52:09.100950  7fcfe27413cbc06f71897ccd5f9ff4b1083cfbbe 192.168.51.41:6380@16380 slave 756d2c3a79c1dfba5df86ccff020e0a72475a47d 0 1760140328554 7 connected
I1010 23:52:09.100926  dad564b61f8758c0d29328b5803edcdaa0811c6f 127.0.0.1:6380@16380 slave a6126962691ab37568773919497b6498254f26fd 1760140328152 1760140328000 13 disconnected
I1010 23:52:09.100943  a6126962691ab37568773919497b6498254f26fd 192.168.132.112:6380@16380 master - 0 1760140328857 13 connected 0-5461
I1010 23:52:09.100954 
I1010 23:52:09.165598  18:M PONG contains mismatching sender ID. About node dad564b61f8758c0d29328b5803edcdaa0811c6f () in shard bec51287c31b192c3a4a1936493523ca0882f87d added 1533203 ms ago, having flags 3074
I1010 23:52:10.109863  7fcfe27413cbc06f71897ccd5f9ff4b1083cfbbe 192.168.51.41:6380@16380 slave 756d2c3a79c1dfba5df86ccff020e0a72475a47d 0 1760140329868 7 connected
I1010 23:52:10.109853  d933a2d1b785fa934f1a5efae9243a937b522ea1 127.0.0.1:6380@16380 myself,master - 0 0 9 connected 10923-16383
I1010 23:52:10.109374  18:M Address updated for node dad564b61f8758c0d29328b5803edcdaa0811c6f (), now 127.0.0.1:6380
I1010 23:52:10.109856  a6126962691ab37568773919497b6498254f26fd 192.168.132.112:6380@16380 master - 0 1760140330069 13 connected 0-5461
I1010 23:52:10.109860  756d2c3a79c1dfba5df86ccff020e0a72475a47d 192.168.37.165:6380@16380 master - 0 1760140329565 7 connected 5462-10922
I1010 23:52:10.109867 
I1010 23:52:10.109824  cba6f96b9c5d2cf97ff2112a77967a0c982873be 192.168.37.138:6380@16380 slave d933a2d1b785fa934f1a5efae9243a937b522ea1 1760140330068 1760140328152 9 connected
I1010 23:52:10.109847  dad564b61f8758c0d29328b5803edcdaa0811c6f 127.0.0.1:6380@16380 slave a6126962691ab37568773919497b6498254f26fd 1760140328152 1760140328000 13 disconnected
I1010 23:52:10.176490  18:M PONG contains mismatching sender ID. About node dad564b61f8758c0d29328b5803edcdaa0811c6f () in shard bec51287c31b192c3a4a1936493523ca0882f87d added 1534214 ms ago, having flags 3074
E1010 23:52:10.261057       1 redisnode.go:448] Failover error: FailoverError: failed to contact replica node when checking progress
E1010 23:52:10.261026       1 admin.go:1022] Failed to get node at 192.168.37.138:6380: unable to retrieve Node Info for the node with address 192.168.37.138:6380 due to error failed to perform action ["CLUSTER" "NODES"], err: read tcp 192.168.5.52:43784->192.168.37.138:6380: i/o timeout
E1010 23:52:10.261245       1 redisnode.go:170] error happened during fail over and forget: cannot failover by address 192.168.5.52:6380: FailoverError: failed to contact replica node when checking progress
I1010 23:52:10.261258       1 main.go:42] fail over done, shut down
I1010 23:52:10.261291       1 main.go:66] redisnode.Run() stopped gracefully
I1010 23:52:10.261280       1 redisnode.go:157] received Stop Signal...

BR,
Seb.

[hpatro]

@esebole Could you also share what's the environment and engine version details?

[esebole]

Hi @hpatro, test was executed with Valkey 8.0.5. The details about environment might be tricky as this is internal env - multiple internal frameworks, wrappers, etc. Please let me know what specific details you would need and I will do my best to dig them out and share.

In my personal opinion I doubt that environment has any influence as this issue is visible only in this one single test case but the environment runs multiple test every single day. I think that this might be somehow related to the cluster configuration. We expect that every single node will operate on the same port 6380. In such case when localhost IP is assigned from whatever reason the pod tries to communicate with self.

19:S Connecting to PRIMARY 127.0.0.1:6380
19:S PRIMARY <-> REPLICA sync started
19:S Non blocking connect for SYNC fired the event.
19:S Primary did not respond to command during SYNC handshake
19:S Failed to read response from the server: (null)

And like how other nodes can detect a failed node in such case?

BR,
Seb.

[esebole]

I was able to reproduce it with the lower logging level. Here is the outcome.

In the test around 25-10-28 23:46:39.866 two pimary pods are deleted: "l7l97" and "dbr84". They have replicas alive. Lets focus on the primary pod "l7l97" and its replica pod "lqshd" which failed to failover. The localhost IP address issue was present.

In below printout you can see the cluster setup. The Pod column is the pod id and the ID column is a Valkey node id.

23:37:02.523620  P: 6 Ru: 6 Re: 6 AA: true
23:37:02.523657  Pod   Version    Role ID   IP              M/Slots
23:37:02.523664  l7l97 8.3.0-h027 M    e96a 192.168.59.123  [5462-10922]  *** WILL BE DELETED
23:37:02.523669  jd9zp 8.3.0-h027 M    dc81 192.168.175.143 [10923-16383]
23:37:02.523673  ztvjv 8.3.0-h027 S    0312 192.168.97.101  abc6
23:37:02.523677  jr6kj 8.3.0-h027 S    ff19 192.168.97.127  dc81
23:37:02.523680  lqshd 8.3.0-h027 S    945c 192.168.142.191 e96a
23:37:02.523685  dbr84 8.3.0-h027 M    abc6 192.168.142.136 [0-5461]  *** WILL BE DELETED
...
23:46:45.318944  P: 5 Ru: 5 Re: 5 AA: false
23:46:45.318977  Pod   Version    Role ID   IP              M/Slots       Worker
23:46:45.318982  jd9zp 8.3.0-h027 M    dc81 192.168.175.143 [10923-16383]
23:46:45.318986  ztvjv 8.3.0-h027 S    0312 192.168.97.101  abc6  *** WILL FAILOVER
23:46:45.318989  jr6kj 8.3.0-h027 S    ff19 192.168.97.127  dc81
23:46:45.318991  lqshd 8.3.0-h027 S    945c 192.168.142.191 e96a  *** EXPECTING TO DO A FAILOVER
23:46:45.318994  dbr84 8.3.0-h027 M    abc6 192.168.142.136 [0-5461]
...

Highlights

In "lqshd" logs we see the change to localhost IP for node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357

23:46:40.111897 20:S node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () announces that it is a replica in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
...
23:46:40.111888 20:S Address updated for node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 (), now 127.0.0.1:6380
...
23:46:40.112434 the node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 has released IP and it is localhost now, but it already failed over and will be deleted. Anyway this address points to self for every other node which is strange...
...
23:46:40.155769 20:S Connecting with Node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 at 127.0.0.1:16380

There was no successful failover but anyway the e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 announced itself as a primary.

23:46:40.157602 20:S node 945c880c005a42a8bed0e3a9219ca2cdefe64f98 () announces that it is a replica in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:40.157609 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:40.157616 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:40.157622 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:40.157628 20:S --- Processing packet of type pong, 2616 bytes
23:46:40.157634 20:S pong packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
23:46:40.157642 20:S PONG contains mismatching sender ID. About node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a added 687862 ms ago, having flags 3074
23:46:40.158001 20:S I/O error reading from node link: connection closed
23:46:40.158708 20:S --- Processing packet of type ping, 2616 bytes
23:46:40.158741 20:S ping packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:40.158747 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d

In a meantime there is another IP update, this time for "e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33" which starts to

23:46:40.158753 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:40.158913 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:40.158922 20:S GOSSIP e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 192.168.59.123:6380@16380 master
23:46:40.197670 20:S --- Processing packet of type ping, 2616 bytes
23:46:40.197702 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:40.197709 20:S Address updated for node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 (), now 127.0.0.1:6380
23:46:40.197718 20:M Connection with primary lost.
23:46:40.197726 20:M Caching the disconnected primary state.
23:46:40.197733 20:S Connecting to PRIMARY 127.0.0.1:6380
23:46:40.197739 20:S PRIMARY <-> REPLICA sync started
23:46:40.197754 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
...
23:46:45.488171 ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave dc81be9ba4688a18d123dca363b94ba4815e0e01 0 1761608805462 7 connected
23:46:45.488203 945c880c005a42a8bed0e3a9219ca2cdefe64f98 127.0.0.1:6380@16380 myself,slave e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 0 0 11 connected
23:46:45.488208 e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 :0@0 master,noaddr - 1761608805454 1761608800052 11 disconnected 5462-10922
23:46:45.488211 dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master - 0 1761608805462 7 connected 10923-16383
23:46:45.488215 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master - 0 1761608805464 15 connected 0-5461
23:46:45.488218 abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 127.0.0.1:6380@16380 slave 03124dd2ecbc25c4b96af5b29ae56d234b865c86 1761608800152 1761608799144 15 disconnected

Logs from Primary pod l7l97

 23:46:39.691058 :142] 20:M Client closed connection id=321 addr=127.0.0.1:43124 laddr=127.0.0.1:6380 fd=80 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=slowlog|get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=252 tot-net-out=20241 tot-cmds=8
 23:46:39.978472 :30] Signal received: terminated
 23:46:39.978532 :38] Shutdown handler called
 23:46:39.978704       1 redisnode.go:325] http server stop error: http: Server closed
 23:46:40.009453       1 admin.go:321] Failover by address: 192.168.59.123:6380
 23:46:40.017796       1 admin.go:345] Starting failover for node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
...
 23:46:40.060578 :142] 20:M Accepted 192.168.59.123:52568
...
 23:46:40.068756 :178] 20:M --- Processing packet of type auth-req, 2256 bytes
 23:46:40.068781 :174] 20:M Failover auth granted to 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () for epoch 15
 23:46:40.073482 :389] Failover command successfully sent to 945c880c005a42a8bed0e3a9219ca2cdefe64f98
 23:46:40.073659 :174] 20:M Manual failover requested by replica 945c880c005a42a8bed0e3a9219ca2cdefe64f98 ().
 23:46:40.073516 :397] Starting to wait for slot transfer from master=(e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33, 192.168.59.123:6380) to replica=(945c880c005a42a8bed0e3a9219ca2cdefe64f98, 192.168.142.191:6380) to complete, total slots to transfer: 5461
 23:46:40.073642 :178] 20:M --- Processing packet of type mfstart, 2256 bytes
 23:46:40.078525 :178] 20:M node 945c880c005a42a8bed0e3a9219ca2cdefe64f98 () announces that it is a replica in shard 0ab48b544cfad7f305561e5b77f883767a118f74
 23:46:40.078543 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:40.078552 :178] 20:M --- Processing packet of type auth-req, 2256 bytes
 23:46:40.078519 :178] 20:M pong packet received: 945c880c005a42a8bed0e3a9219ca2cdefe64f98
 23:46:40.078556 :174] 20:M Failover auth granted to 945c880c005a42a8bed0e3a9219ca2cdefe64f98 () for epoch 16
 23:46:40.078537 :178] 20:M GOSSIP abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 192.168.142.136:6380@16380 master
 23:46:40.078492 :178] 20:M --- Processing packet of type pong, 2616 bytes
 23:46:40.078548 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:40.082465 :417] Waiting for failover to complete slot transfer...
 23:46:40.083220 :178] 20:M --- Processing packet of type auth-req, 2256 bytes
 23:46:40.083238 :176] 20:M Failover auth denied to 945c880c005a42a8bed0e3a9219ca2cdefe64f98 : can't vote about this primary before 3991 milliseconds
 23:46:40.083312 :178] 20:M pong packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
 23:46:40.083814 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:40.083825 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:40.083306 :178] 20:M --- Processing packet of type pong, 2512 bytes
 23:46:40.083316 :178] 20:M node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
 23:46:40.094278 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:40.094307 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:40.094994 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:40.099254 :178] 20:M --- Processing packet of type pong, 2616 bytes
 23:46:40.099302 :178] 20:M GOSSIP abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 192.168.142.136:6380@16380 master
 23:46:40.094217 :178] 20:M ping packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
 23:46:40.094945 :178] 20:M --- Processing packet of type pong, 2616 bytes
 23:46:40.094980 :178] 20:M node 945c880c005a42a8bed0e3a9219ca2cdefe64f98 () announces that it is a replica in shard 0ab48b544cfad7f305561e5b77f883767a118f74
 23:46:40.094988 :178] 20:M GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
 23:46:40.095265 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:40.099283 :178] 20:M node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
 23:46:40.094199 :178] 20:M --- Processing packet of type ping, 2616 bytes
 23:46:40.094296 :178] 20:M GOSSIP abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 192.168.142.136:6380@16380 master
 23:46:40.095225 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:40.095257 :174] 20:M Node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () is now a replica of node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
 23:46:40.099296 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:40.094272 :178] 20:M node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
 23:46:40.094973 :178] 20:M pong packet received: 945c880c005a42a8bed0e3a9219ca2cdefe64f98
 23:46:40.095236 :178] 20:M --- Processing packet of type pong, 2616 bytes
 23:46:40.095243 :178] 20:M pong packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
 23:46:40.095250 :178] 20:M node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () announces that it is a replica in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
 23:46:40.095273 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:40.095280 :178] 20:M GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
 23:46:40.099275 :178] 20:M pong packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
 23:46:40.099290 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:40.111332 :178] 20:M ping packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
 23:46:40.111381 :174] 20:M Address updated for node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 (), now 127.0.0.1:6380
 23:46:40.111311 :178] 20:M --- Processing packet of type ping, 2616 bytes
 23:46:40.115489 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:40.115495 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:40.115506 :142] 20:M Accepted 127.0.0.1:43138
 23:46:40.115405 :344] 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master - 0 1761608800099 15 connected 0-5461
 23:46:40.115434 :344] dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master - 0 1761608799185 7 connected 10923-16383
 23:46:40.115445 :344] 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 0 1761608800094 11 connected
 23:46:40.115502 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:40.115429 :344] e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 127.0.0.1:6380@16380 myself,master - 0 0 11 connected 5462-10922
 23:46:40.115448 :344]
 23:46:40.115478 :178] 20:M node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () announces that it is a replica in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
 23:46:40.115438 :344] abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 127.0.0.1:6380@16380 slave 03124dd2ecbc25c4b96af5b29ae56d234b865c86 0 1761608800094 15 disconnected
 23:46:40.115442 :344] ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave dc81be9ba4688a18d123dca363b94ba4815e0e01 0 1761608799287 7 connected
 23:46:40.181332 :142] 20:M Accepted 192.168.142.144:38038
 23:46:40.186730 :142] 20:M Reading from client: error:0A000126:SSL routines::unexpected eof while reading
 23:46:40.186520 :142] 20:M Reading from client: error:0A000126:SSL routines::unexpected eof while reading
 23:46:40.199825 :178] 20:M node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
 23:46:40.199829 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:40.199811 :178] 20:M Connecting with Node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 at 127.0.0.1:16380
 23:46:40.199821 :178] 20:M pong packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
 23:46:40.199832 :178] 20:M --- Processing packet of type pong, 2512 bytes
 23:46:40.199836 :178] 20:M pong packet received: 945c880c005a42a8bed0e3a9219ca2cdefe64f98
 23:46:40.199844 :178] 20:M GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
 23:46:40.199798 :142] 20:M Client closed connection id=172 addr=192.168.142.191:48008 laddr=192.168.59.123:6380 fd=63 name=*redacted* age=688 idle=0 flags=S db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=1 omem=20504 tot-mem=42904 events=r cmd=replconf user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=29601 tot-net-out=652595996 tot-cmds=690
 23:46:40.199816 :178] 20:M --- Processing packet of type pong, 2408 bytes
 23:46:40.199805 :174] 20:M Connection with replica 192.168.142.191:6380 lost.
 23:46:40.199760 :142] 20:M Accepting cluster node connection from 127.0.0.1:48614
 23:46:40.199787 :178] 20:M I/O error reading from node link: connection closed
 23:46:40.199793 :178] 20:M I/O error reading from node link: connection closed
 23:46:40.199840 :178] 20:M node 945c880c005a42a8bed0e3a9219ca2cdefe64f98 () announces that it is a replica in shard 0ab48b544cfad7f305561e5b77f883767a118f74
 23:46:40.199876 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:40.199861 :178] 20:M node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
 23:46:40.199879 :178] 20:M --- Processing packet of type pong, 2616 bytes
 23:46:40.199884 :178] 20:M pong packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
 23:46:40.199852 :178] 20:M --- Processing packet of type ping, 2512 bytes
 23:46:40.199856 :178] 20:M ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
 23:46:40.199889 :174] 20:M PONG contains mismatching sender ID. About node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a added 1507402 ms ago, having flags 3074
 23:46:40.199866 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:40.199848 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:40.199897 :178] 20:M I/O error reading from node link: connection closed
 23:46:40.296475 :178] 20:M Pinging node ff19d2219d76a6d8959424b4de4a025b1f3632e6
 23:46:40.296836 :178] 20:M I/O error reading from node link: connection closed
 23:46:40.297214 :178] 20:M GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
 23:46:40.297203 :178] 20:M pong packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
 23:46:40.297185 :178] 20:M --- Processing packet of type pong, 2408 bytes
 23:46:40.297209 :178] 20:M node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
 23:46:41.100350 :178] 20:M ping packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
 23:46:41.100356 :178] 20:M node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
 23:46:41.100366 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:41.100362 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:41.100371 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:41.100317 :178] 20:M --- Processing packet of type ping, 2616 bytes
 23:46:41.106856 :178] 20:M I/O error reading from node link: connection closed
 23:46:41.107212 :178] 20:M pong packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
 23:46:41.107216 :178] 20:M node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
 23:46:41.107198 :178] 20:M --- Processing packet of type pong, 2616 bytes
 23:46:41.107220 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:41.107329 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:41.107318 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:41.119627 :344] dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master - 0 1761608800197 7 connected 10923-16383
 23:46:41.119638 :344] ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave dc81be9ba4688a18d123dca363b94ba4815e0e01 0 1761608800297 7 connected
 23:46:41.118998 :178] 20:M --- Processing packet of type ping, 2616 bytes
 23:46:41.119040 :174] 20:M Address updated for node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 (), now 127.0.0.1:6380
 23:46:41.119051 :178] 20:M node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () announces that it is a replica in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
 23:46:41.119059 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:41.119030 :178] 20:M ping packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
 23:46:41.119068 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:41.119648 :344] 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 1761608801103 1761608800198 11 connected
 23:46:41.119653 :344]
 23:46:41.119082 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:41.119587 :344] 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master - 0 1761608801107 15 connected 0-5461
 23:46:41.119613 :344] e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 127.0.0.1:6380@16380 myself,master - 0 0 11 connected 5462-10922
 23:46:41.119632 :344] abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 127.0.0.1:6380@16380 slave 03124dd2ecbc25c4b96af5b29ae56d234b865c86 1761608800197 1761608800094 15 disconnected
 23:46:41.143353 :142] 20:M Accepted 192.168.142.144:38048
 23:46:41.204814 :142] 20:M Accepting cluster node connection from 127.0.0.1:48630
 23:46:41.206791 :178] 20:M Connecting with Node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 at 127.0.0.1:16380
 23:46:41.206914 :178] 20:M pong packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
 23:46:41.206896 :178] 20:M --- Processing packet of type pong, 2408 bytes
 23:46:41.206925 :178] 20:M node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
 23:46:41.206934 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:41.208007 :178] 20:M ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
 23:46:41.208132 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:41.208146 :174] 20:M PONG contains mismatching sender ID. About node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a added 1508411 ms ago, having flags 3074
 23:46:41.207989 :178] 20:M --- Processing packet of type ping, 2616 bytes
 23:46:41.208136 :178] 20:M --- Processing packet of type pong, 2616 bytes
 23:46:41.208140 :178] 20:M pong packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
 23:46:41.208118 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:41.208128 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:41.208011 :178] 20:M node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
 23:46:41.208370 :178] 20:M I/O error reading from node link: connection closed
 23:46:41.306049 :178] 20:M Pinging node ff19d2219d76a6d8959424b4de4a025b1f3632e6
 23:46:41.306917 :178] 20:M node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
 23:46:41.306875 :178] 20:M --- Processing packet of type pong, 2408 bytes
 23:46:41.306906 :178] 20:M pong packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
 23:46:41.306929 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:41.380730 :142] 20:M Accepted 192.168.175.142:50670
 23:46:41.385146 :142] 20:M Reading from client: error:0A000126:SSL routines::unexpected eof while reading
 23:46:41.385290 :142] 20:M Reading from client: error:0A000126:SSL routines::unexpected eof while reading
 23:46:41.761963 :142] 20:M Reading from client: error:0A000126:SSL routines::unexpected eof while reading
 23:46:41.829271 :142] 20:M Reading from client: error:0A000126:SSL routines::unexpected eof while reading
 23:46:42.112738 :178] 20:M --- Processing packet of type pong, 2616 bytes
 23:46:42.112800 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:42.112778 :178] 20:M pong packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
 23:46:42.112787 :178] 20:M node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
 23:46:42.112793 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:42.112805 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:42.127773 :178] 20:M ping packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
 23:46:42.127745 :178] 20:M --- Processing packet of type ping, 2616 bytes
 23:46:42.127786 :178] 20:M node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () announces that it is a replica in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
 23:46:42.127790 :178] 20:M GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
 23:46:42.127780 :174] 20:M Address updated for node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 (), now 127.0.0.1:6380
 23:46:42.127795 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:42.127800 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:42.128135 :344] ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave dc81be9ba4688a18d123dca363b94ba4815e0e01 0 1761608801306 7 connected
 23:46:42.128142 :344]
 23:46:42.128131 :344] abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 127.0.0.1:6380@16380 slave 03124dd2ecbc25c4b96af5b29ae56d234b865c86 1761608800197 1761608800094 15 disconnected
 23:46:42.128138 :344] 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 1761608802112 1761608800198 11 connected
 23:46:42.128127 :344] dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master - 0 1761608801206 7 connected 10923-16383
 23:46:42.128099 :344] 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master - 0 1761608802112 15 connected 0-5461
 23:46:42.128122 :344] e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 127.0.0.1:6380@16380 myself,master - 0 0 11 connected 5462-10922
 23:46:42.216713 :178] 20:M GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
 23:46:42.216754 :178] 20:M ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
 23:46:42.216759 :178] 20:M node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
 23:46:42.216774 :178] 20:M --- Processing packet of type pong, 2616 bytes
 23:46:42.216693 :178] 20:M --- Processing packet of type pong, 2512 bytes
 23:46:42.216705 :178] 20:M node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
 23:46:42.216710 :178] 20:M GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
 23:46:42.216763 :178] 20:M GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
 23:46:42.216766 :178] 20:M GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave
 23:46:42.216783 :178] 20:M pong packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
 23:46:42.216788 :174] 20:M PONG contains mismatching sender ID. About node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a added 1509419 ms ago, having flags 3074
 23:46:42.216661 :142] 20:M Accepting cluster node connection from 127.0.0.1:48644
 23:46:42.216700 :178] 20:M pong packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
 23:46:42.216746 :178] 20:M Connecting with Node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 at 127.0.0.1:16380
 23:46:42.216750 :178] 20:M --- Processing packet of type ping, 2616 bytes
 23:46:42.216770 :178] 20:M GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
 23:46:42.217132 :178] 20:M I/O error reading from node link: connection closed
 23:46:42.283829 :170] error happened during fail over and forget: cannot failover by address 192.168.59.123:6380: FailoverError: failed to contact replica node when checking progress
...
 23:46:42.283891 :157] received Stop Signal...
 23:46:42.283630 :448] Failover error: FailoverError: failed to contact replica node when checking progress
 23:46:42.283906 :66] redisnode.Run() stopped gracefully

Logs from Replica pod 'lqshd':

23:35:11.049903 20:C WARNING: Changing databases number from 16 to 1 since we are in cluster mode
23:35:11.049965 20:C oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
23:35:11.049980 20:C Valkey version=8.0.5, bits=64, commit=00000000, modified=0, pid=20, just started
23:35:11.049986 20:C Configuration loaded
23:35:11.050335 20:M monotonic clock: POSIX clock_gettime
23:35:11.050811 20:M Running mode=cluster, port=6380.
23:35:11.050848 20:M WARNING: The TCP backlog setting of 65536 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 4096.
23:35:11.051227 20:M No cluster configuration found, I'm 945c880c005a42a8bed0e3a9219ca2cdefe64f98
23:35:11.056769 20:M Server initialized
23:35:11.056778 20:M The AOF directory appendonlydir doesn't exist
23:35:11.056812 20:M Ready to accept connections tls
23:35:11.057145 20:M Total: 0 clients connected (0 replicas), 2690952 (2.57M) bytes in use
23:35:11.089081 20:M Accepted 127.0.0.1:52188
23:35:11.093335 20:M Client closed connection id=3 addr=127.0.0.1:52188 laddr=127.0.0.1:6380 fd=13 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=ping user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=80 tot-net-out=12 tot-cmds=2
23:35:11.093446 20:M Accepted 127.0.0.1:52194
23:35:12.126338 20:M Accepted 192.168.59.124:45058
23:35:12.198897 20:M Client closed connection id=5 addr=192.168.59.124:45058 laddr=192.168.142.191:6380 fd=14 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=0 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=133 tot-net-out=498 tot-cmds=3
23:35:12.207660 20:M Accepted 192.168.59.124:45074
23:35:12.272564 20:M configEpoch set to 12 via CLUSTER SET-CONFIG-EPOCH
23:35:12.272873 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 myself,master - 0 0 12 connected
...
23:46:39.081792 20:S --- Processing packet of type ping, 2616 bytes
23:46:39.081808 20:S ping packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:39.081813 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a replica in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:39.081817 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:39.081823 20:S GOSSIP e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 192.168.59.123:6380@16380 master
23:46:39.081827 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:39.084295 20:S --- Processing packet of type ping, 2616 bytes
23:46:39.084320 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:39.084325 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:39.084330 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:39.084335 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 slave
23:46:39.084438 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:39.096937 20:S --- Processing packet of type ping, 2616 bytes
23:46:39.096960 20:S ping packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:39.096965 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:39.096969 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 slave
23:46:39.097070 20:S GOSSIP abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 192.168.142.136:6380@16380 master
23:46:39.097085 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
...
23:46:40.073350 20:S Manual failover user request accepted (user request from 'id=143 addr=192.168.59.123:33792 laddr=192.168.142.191:6380 fd=47 name=*redacted* age=1 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=15 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17311 events=r cmd=cluster|failover user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=288 tot-net-out=3012 tot-cmds=6').
23:46:40.073773 20:S --- Processing packet of type ping, 2616 bytes
23:46:40.073784 20:S Received replication offset for paused primary manual failover: 1309757490
23:46:40.073790 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:40.073865 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:40.073869 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:40.073873 20:S GOSSIP abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 192.168.142.136:6380@16380 master
23:46:40.073876 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:40.073880 20:S All primary replication stream processed, manual failover can start.
23:46:40.073885 20:S Start of election delayed for 0 milliseconds (rank #0, offset 1309757490).
23:46:40.073890 20:S Starting a failover election for epoch 16, node config epoch is 11
23:46:40.078559 20:S --- Processing packet of type pong, 2616 bytes
23:46:40.078581 20:S pong packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:40.078589 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:40.079079 20:S GOSSIP abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 192.168.142.136:6380@16380 master
23:46:40.079104 20:S GOSSIP e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 192.168.59.123:6380@16380 master
23:46:40.079110 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:40.079264 20:S --- Processing packet of type auth-ack, 2256 bytes
23:46:40.079277 20:S Forced failover user request accepted (user request from 'id=141 addr=192.168.59.124:52094 laddr=192.168.142.191:6380 fd=45 name=*redacted* age=1 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=20 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17324 events=r cmd=cluster|failover user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=235 tot-net-out=3019 tot-cmds=5').
23:46:40.079286 20:S Failover election in progress for epoch 16, but received a new manual failover. Resetting the election.
23:46:40.079296 20:S Start of election delayed for 0 milliseconds (rank #0, offset 1309757490).
23:46:40.082263 20:S --- Processing packet of type auth-ack, 2256 bytes
23:46:40.082281 20:S --- Processing packet of type auth-ack, 2256 bytes
23:46:40.082331 20:S Starting a failover election for epoch 17, node config epoch is 11
23:46:40.084812 20:S --- Processing packet of type auth-ack, 2256 bytes
23:46:40.084841 20:S Received replication offset for paused primary manual failover: 1309757490
23:46:40.084852 20:S Currently unable to failover: Waiting for votes, but majority still not reached.
23:46:40.084860 20:S Needed quorum: 2. Number of votes received so far: 1
23:46:40.094286 20:S --- Processing packet of type ping, 2616 bytes
23:46:40.094309 20:S ping packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:40.094323 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:40.094329 20:S GOSSIP e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 192.168.59.123:6380@16380 master
23:46:40.094336 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:40.094342 20:S GOSSIP abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 192.168.142.136:6380@16380 master
23:46:40.094883 20:S --- Processing packet of type ping, 2616 bytes
23:46:40.094904 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:40.094911 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:40.094917 20:S GOSSIP abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 192.168.142.136:6380@16380 master
23:46:40.094925 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:40.094930 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:40.105767 20:S --- Processing packet of type ping, 2616 bytes
23:46:40.105790 20:S ping packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:40.105798 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:40.105803 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:40.105809 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:40.105817 20:S GOSSIP e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 192.168.59.123:6380@16380 master
23:46:40.111847 20:S --- Processing packet of type ping, 2616 bytes
23:46:40.111879 20:S ping packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
23:46:40.111888 20:S Address updated for node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 (), now 127.0.0.1:6380
23:46:40.111897 20:S node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () announces that it is a replica in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:40.111906 20:S Node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () is now a replica of node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:40.111913 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:40.111921 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:40.111928 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:40.112377 ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave dc81be9ba4688a18d123dca363b94ba4815e0e01 0 1761608799850 7 connected
23:46:40.112410 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 myself,slave e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 0 0 11 connected
23:46:40.112418 e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 192.168.59.123:6380@16380 master - 0 1761608800052 11 connected 5462-10922
23:46:40.112423 dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master - 0 1761608799144 7 connected 10923-16383
23:46:40.112429 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master - 0 1761608800053 15 connected 0-5461
23:46:40.112434 abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 127.0.0.1:6380@16380 slave 03124dd2ecbc25c4b96af5b29ae56d234b865c86 0 1761608799144 15 disconnected
23:46:40.152926 20:S Accepting cluster node connection from 127.0.0.1:42122
23:46:40.154123 20:S --- Processing packet of type pong, 2408 bytes
23:46:40.154159 20:S pong packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:40.154168 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:40.154174 20:S GOSSIP e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 192.168.59.123:6380@16380 master
23:46:40.155769 20:S Connecting with Node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 at 127.0.0.1:16380
23:46:40.157323 20:S --- Processing packet of type ping, 2616 bytes
23:46:40.157344 20:S ping packet received: 945c880c005a42a8bed0e3a9219ca2cdefe64f98
23:46:40.157351 20:S node 945c880c005a42a8bed0e3a9219ca2cdefe64f98 () announces that it is a replica in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:40.157359 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:40.157560 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:40.157568 20:S GOSSIP e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 192.168.59.123:6380@16380 master
23:46:40.157574 20:S --- Processing packet of type ping, 2616 bytes
23:46:40.157597 20:S ping packet received: 945c880c005a42a8bed0e3a9219ca2cdefe64f98
23:46:40.157602 20:S node 945c880c005a42a8bed0e3a9219ca2cdefe64f98 () announces that it is a replica in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:40.157609 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:40.157616 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:40.157622 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:40.157628 20:S --- Processing packet of type pong, 2616 bytes
23:46:40.157634 20:S pong packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
23:46:40.157642 20:S PONG contains mismatching sender ID. About node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a added 687862 ms ago, having flags 3074
23:46:40.158001 20:S I/O error reading from node link: connection closed
23:46:40.158708 20:S --- Processing packet of type ping, 2616 bytes
23:46:40.158741 20:S ping packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:40.158747 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:40.158753 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:40.158913 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:40.158922 20:S GOSSIP e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 192.168.59.123:6380@16380 master
23:46:40.197670 20:S --- Processing packet of type ping, 2616 bytes
23:46:40.197702 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:40.197709 20:S Address updated for node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 (), now 127.0.0.1:6380
23:46:40.197718 20:M Connection with primary lost.
23:46:40.197726 20:M Caching the disconnected primary state.
23:46:40.197733 20:S Connecting to PRIMARY 127.0.0.1:6380
23:46:40.197739 20:S PRIMARY <-> REPLICA sync started
23:46:40.197754 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:40.197763 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:40.197770 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:40.197778 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:40.197784 20:S Accepted 127.0.0.1:49552
23:46:40.199641 20:S Non blocking connect for SYNC fired the event.
23:46:45.456976 20:S Failed to read response from the server: (null)
23:46:45.457036 20:S Primary did not respond to command during SYNC handshake
23:46:45.457048 20:S Manual failover timed out.
23:46:45.457055 20:S Client closed connection id=144 addr= laddr=127.0.0.1:6380 fd=22 name=*redacted* age=5 idle=5 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=ping user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=14 tot-net-out=7 tot-cmds=1
23:46:45.457065 20:S --- Processing packet of type ping, 2616 bytes
23:46:45.457075 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457082 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457089 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.457096 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457103 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457110 20:S --- Processing packet of type ping, 2408 bytes
23:46:45.457117 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457123 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457136 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457143 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457153 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457160 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457167 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457173 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457179 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457185 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457190 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457196 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457202 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.457207 20:S --- Processing packet of type ping, 2408 bytes
23:46:45.457213 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457225 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457231 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457238 20:S --- Processing packet of type ping, 2616 bytes
23:46:45.457243 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457249 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457257 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.457264 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457270 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457286 20:S --- Processing packet of type ping, 2616 bytes
23:46:45.457292 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457300 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457306 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457312 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457318 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.457329 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457335 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457340 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457345 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457352 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457357 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457362 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457369 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457375 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457382 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457390 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457397 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457408 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457416 20:S GOSSIP abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 127.0.0.1:6380@16380 slave
23:46:45.457422 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457428 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457434 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457440 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457446 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457452 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457457 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457464 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457469 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457475 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457481 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.457487 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457495 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457509 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457515 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457520 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457526 20:S --- Processing packet of type ping, 2408 bytes
23:46:45.457532 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457539 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457545 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457551 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457557 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457564 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457569 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457575 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.457581 20:S --- Processing packet of type ping, 2408 bytes
23:46:45.457587 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457594 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457601 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.457606 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457613 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457620 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457627 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457639 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457647 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457653 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457660 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457665 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457672 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.457677 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457684 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457691 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457702 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457707 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457714 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457720 20:S ping packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.457728 20:S node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () announces that it is a primary in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.457734 20:S GOSSIP abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 127.0.0.1:6380@16380 slave
23:46:45.457740 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.457746 20:S I/O error reading from node link: error:0A000126:SSL routines::unexpected eof while reading
23:46:45.457752 20:S --- Processing packet of type ping, 2616 bytes
23:46:45.457757 20:S ping packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:45.457765 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:45.457771 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457777 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457784 20:S GOSSIP e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 192.168.59.123:6380@16380 master
23:46:45.457791 20:S I/O error reading from node link: connection closed
23:46:45.457797 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457803 20:S ping packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:45.457809 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:45.457817 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.457822 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.457831 20:S I/O error reading from node link: connection closed
23:46:45.457842 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.457848 20:S ping packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
23:46:45.457856 20:S Address updated for node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 (), now 127.0.0.1:6380
23:46:45.457865 20:S node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () announces that it is a replica in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:45.457873 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.457886 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.457892 20:S I/O error reading from node link: connection closed
23:46:45.488171 ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave dc81be9ba4688a18d123dca363b94ba4815e0e01 0 1761608805462 7 connected
23:46:45.488203 945c880c005a42a8bed0e3a9219ca2cdefe64f98 127.0.0.1:6380@16380 myself,slave e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 0 0 11 connected
23:46:45.488208 e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 :0@0 master,noaddr - 1761608805454 1761608800052 11 disconnected 5462-10922
23:46:45.488211 dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master - 0 1761608805462 7 connected 10923-16383
23:46:45.488215 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master - 0 1761608805464 15 connected 0-5461
23:46:45.488218 abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 127.0.0.1:6380@16380 slave 03124dd2ecbc25c4b96af5b29ae56d234b865c86 1761608800152 1761608799144 15 disconnected
23:46:45.488222 
23:46:45.488555 20:S --- Processing packet of type ping, 2304 bytes
23:46:45.488567 20:S ping packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:45.488572 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:45.488576 20:S I/O error reading from node link: connection closed
23:46:45.488580 20:S Accepted 127.0.0.1:49568
23:46:45.488584 20:S Error accepting a client connection: error:0A000126:SSL routines::unexpected eof while reading (addr= laddr=127.0.0.1:6380)
23:46:45.488590 20:S Accepting cluster node connection from 192.168.175.143:51396
23:46:45.488594 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:45.488599 20:S Client closed connection id=118 addr=192.168.97.86:45294 laddr=192.168.142.191:6380 fd=42 name=*redacted* age=222 idle=82 flags=tB db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22424 events=r cmd=ping user=*redacted* redir=0 resp=3 lib-name=rueidis lib-ver=1.0.62 tot-net-in=329 tot-net-out=208 tot-cmds=11
23:46:45.488603 20:S Client closed connection id=143 addr=192.168.59.123:33792 laddr=192.168.142.191:6380 fd=47 name=*redacted* age=6 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=cluster|nodes user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=376 tot-net-out=5339 tot-cmds=10
23:46:45.488607 20:S Client closed connection id=141 addr=192.168.59.124:52094 laddr=192.168.142.191:6380 fd=45 name=*redacted* age=6 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=cluster|nodes user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=323 tot-net-out=5308 tot-cmds=9
23:46:45.488610 20:S Client closed connection id=142 addr=192.168.142.136:37448 laddr=192.168.142.191:6380 fd=46 name=*redacted* age=6 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=cluster|nodes user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=225 tot-net-out=2332 tot-cmds=5
23:46:45.488614 20:S Client closed connection id=4 addr=127.0.0.1:52194 laddr=127.0.0.1:6380 fd=13 name=*redacted* age=694 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=7 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=ping user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=5380 tot-net-out=58908 tot-cmds=274
23:46:45.488618 20:S Client closed connection id=57 addr=192.168.59.111:58414 laddr=192.168.142.191:6380 fd=14 name=*redacted* age=586 idle=0 flags=tB db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=7 obl=0 oll=0 omem=0 tot-mem=22472 events=r cmd=ping user=*redacted* redir=0 resp=3 lib-name=rueidis lib-ver=1.0.66 tot-net-in=8414 tot-net-out=7208 tot-cmds=585
23:46:45.488623 20:S Reading from client: error:0A000126:SSL routines::unexpected eof while reading
23:46:45.488634 20:S Accepted 192.168.97.80:40316
23:46:45.488638 20:S Accepting cluster node connection from 192.168.142.136:46804
23:46:45.488642 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:45.488646 20:S --- Processing packet of type pong, 2512 bytes
23:46:45.488655 20:S pong packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:45.488659 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:45.488664 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.488672 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.488677 20:S --- Processing packet of type pong, 2408 bytes
23:46:45.488681 20:S pong packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:45.488685 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:45.488690 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.488693 20:S Error writing to client: Broken pipe
23:46:45.488697 20:S Error writing to client: Broken pipe
23:46:45.488701 20:S Accepted 127.0.0.1:49590
23:46:45.488705 20:S Error accepting a client connection: error:0A000126:SSL routines::unexpected eof while reading (addr= laddr=127.0.0.1:6380)
23:46:45.488714 20:S Accepting cluster node connection from 192.168.97.101:37558
23:46:45.488719 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:45.488724 20:S --- Processing packet of type pong, 2408 bytes
23:46:45.488729 20:S pong packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:45.488733 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:45.488738 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.488742 20:S Accepted 192.168.142.136:37458
23:46:45.488746 20:S Error accepting a client connection: error:0A000126:SSL routines::unexpected eof while reading (addr= laddr=192.168.142.191:6380)
23:46:45.488751 20:S Accepting cluster node connection from 192.168.97.127:38116
23:46:45.488755 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:45.488759 20:S Accepted 192.168.142.136:58482
23:46:45.488763 20:S Accepting cluster node connection from 192.168.175.143:40736
23:46:45.488767 20:S Accepting cluster node connection from 192.168.142.136:46812
23:46:45.488770 20:S Accepting cluster node connection from 192.168.97.101:37564
23:46:45.488774 20:S Accepting cluster node connection from 192.168.97.127:38126
23:46:45.488777 20:S Accepting cluster node connection from 127.0.0.1:47678
23:46:45.488782 20:S Accepted 192.168.59.124:52110
23:46:45.488786 20:S Connecting with Node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 at 127.0.0.1:16380
23:46:45.488795 20:S Accepted 127.0.0.1:36980
23:46:45.488799 20:S --- Processing packet of type ping, 2408 bytes
23:46:45.488803 20:S ping packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:45.488808 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:45.488813 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.488818 20:S --- Processing packet of type ping, 2720 bytes
23:46:45.488822 20:S ping packet received: abc6d2b9e27bfb7329d5c30f51c665e07c1b3357
23:46:45.488826 20:S node abc6d2b9e27bfb7329d5c30f51c665e07c1b3357 () announces that it is a replica in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:45.488830 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.488834 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.488838 20:S GOSSIP e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 192.168.59.123:6380@16380 master,fail?
23:46:45.488843 20:S GOSSIP 945c880c005a42a8bed0e3a9219ca2cdefe64f98 192.168.142.191:6380@16380 slave,fail?
23:46:45.488850 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.488854 20:S ping packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:45.488858 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:45.488861 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.488865 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.488869 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.488873 20:S ping packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:45.488877 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:45.488881 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.488885 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.488889 20:S --- Processing packet of type ping, 2616 bytes
23:46:45.488893 20:S ping packet received: 945c880c005a42a8bed0e3a9219ca2cdefe64f98
23:46:45.488896 20:S node 945c880c005a42a8bed0e3a9219ca2cdefe64f98 () announces that it is a replica in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.488900 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:45.488904 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.488912 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.488915 20:S --- Processing packet of type ping, 2512 bytes
23:46:45.488918 20:S ping packet received: 945c880c005a42a8bed0e3a9219ca2cdefe64f98
23:46:45.488922 20:S node 945c880c005a42a8bed0e3a9219ca2cdefe64f98 () announces that it is a replica in shard 0ab48b544cfad7f305561e5b77f883767a118f74
23:46:45.488927 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:45.488931 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:45.488934 20:S Reading from client: error:0A000126:SSL routines::unexpected eof while reading
23:46:45.488938 20:S --- Processing packet of type pong, 2616 bytes
23:46:45.488943 20:S pong packet received: e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33
23:46:45.488949 20:S PONG contains mismatching sender ID. About node e96a7dc3d12c8595ee939128f5f6cf6a89bc4e33 () in shard 0ab48b544cfad7f305561e5b77f883767a118f74 added 693192 ms ago, having flags 3329
23:46:45.488958 20:S I/O error reading from node link: connection closed
23:46:45.551184 20:S Client closed connection id=149 addr=192.168.142.136:58482 laddr=192.168.142.191:6380 fd=14 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=cluster|forget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=136 tot-net-out=1554 tot-cmds=3
23:46:45.602002 20:S Client closed connection id=150 addr=192.168.59.124:52110 laddr=192.168.142.191:6380 fd=22 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=193 tot-net-out=3079 tot-cmds=5
23:46:45.616981 20:S Accepted 192.168.59.124:52122
23:46:45.670163 20:S Client closed connection id=153 addr=192.168.59.124:52122 laddr=192.168.142.191:6380 fd=13 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=133 tot-net-out=1395 tot-cmds=3
23:46:45.670776 20:S Accepted 192.168.59.124:52138
23:46:45.781883 20:S Client closed connection id=154 addr=192.168.59.124:52138 laddr=192.168.142.191:6380 fd=13 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=133 tot-net-out=1395 tot-cmds=3
23:46:45.782583 20:S Accepted 192.168.59.124:52148
23:46:45.851008 20:S Client closed connection id=155 addr=192.168.59.124:52148 laddr=192.168.142.191:6380 fd=13 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=133 tot-net-out=1395 tot-cmds=3
23:46:45.852805 20:S Accepted 192.168.59.124:52158
23:46:45.906693 20:S Client closed connection id=156 addr=192.168.59.124:52158 laddr=192.168.142.191:6380 fd=13 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=133 tot-net-out=1395 tot-cmds=3
23:46:45.921304 20:S Accepted 192.168.59.124:52168
23:46:45.965905 20:S Client closed connection id=157 addr=192.168.59.124:52168 laddr=192.168.142.191:6380 fd=13 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=133 tot-net-out=1395 tot-cmds=3
23:46:45.966626 20:S Accepted 192.168.59.124:52174
23:46:46.014318 20:S Client closed connection id=158 addr=192.168.59.124:52174 laddr=192.168.142.191:6380 fd=13 name=*redacted* age=1 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=133 tot-net-out=1395 tot-cmds=3
23:46:46.059767 20:S Pinging node ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:46.060217 20:S --- Processing packet of type pong, 2464 bytes
23:46:46.060240 20:S pong packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:46.060246 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:46.060250 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:46.157090 20:S Accepted 192.168.59.124:52182
23:46:46.162151 20:S Connecting to PRIMARY 127.0.0.1:6380
23:46:46.162185 20:S PRIMARY <-> REPLICA sync started
23:46:46.162192 20:S Accepted 127.0.0.1:37090
23:46:46.162365 20:S Non blocking connect for SYNC fired the event.
23:46:51.342276 20:S Failed to read response from the server: (null)
23:46:51.342310 20:S Primary did not respond to command during SYNC handshake
23:46:51.342871 20:S Client closed connection id=161 addr= laddr=127.0.0.1:6380 fd=16 name=*redacted* age=5 idle=5 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=ping user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=14 tot-net-out=7 tot-cmds=1
23:46:51.343168 20:S --- Processing packet of type ping, 2464 bytes
23:46:51.343204 20:S ping packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:51.343213 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:51.343225 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:51.343295 20:S I/O error reading from node link: connection closed
23:46:51.343396 20:S --- Processing packet of type ping, 2464 bytes
23:46:51.343414 20:S ping packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:51.343456 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:51.343463 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:51.343468 20:S I/O error reading from node link: connection closed
23:46:51.343557 20:S --- Processing packet of type ping, 2568 bytes
23:46:51.343589 20:S ping packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:51.343596 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:51.343600 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:51.343674 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:51.343680 20:S I/O error reading from node link: connection closed
23:46:51.343718 20:S Accepted 192.168.59.111:45036
23:46:51.345386 20:S Accepting cluster node connection from 192.168.97.127:38138
23:46:51.346226 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:51.346538 20:S Client closed connection id=160 addr=192.168.59.124:52182 laddr=192.168.142.191:6380 fd=13 name=*redacted* age=5 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=cluster|nodes user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=101 tot-net-out=676 tot-cmds=2
23:46:51.346653 20:S Accepted 192.168.142.171:47710
23:46:51.348048 20:S Accepting cluster node connection from 192.168.175.143:40746
23:46:51.348765 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:51.348832 20:S Reading from client: error:0A000126:SSL routines::unexpected eof while reading
23:46:51.350783 20:S Accepted 127.0.0.1:37364
23:46:51.351813 20:S Accepting cluster node connection from 192.168.97.101:37566
23:46:51.352538 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:51.352709 20:S Accepted 192.168.97.86:56090
23:46:51.353455 20:S Reading from client: error:0A000126:SSL routines::unexpected eof while reading
23:46:51.353584 20:S Accepting cluster node connection from 192.168.97.127:38152
23:46:51.356362       1 redisnode.go:310] liveness ok
23:46:51.356435 20:S Accepting cluster node connection from 192.168.175.143:40758
23:46:51.356449 20:S Accepting cluster node connection from 192.168.97.101:37576
23:46:51.356456 20:S --- Processing packet of type pong, 2464 bytes
23:46:51.356464 20:S pong packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:51.356471 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:51.356477 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:51.356484 20:S --- Processing packet of type pong, 2464 bytes
23:46:51.356491 20:S pong packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:51.356496 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:51.356502 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:51.356509 20:S --- Processing packet of type pong, 2568 bytes
23:46:51.356516 20:S pong packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:51.356522 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:51.356528 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:51.356534 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:51.361713 20:S --- Processing packet of type ping, 2464 bytes
23:46:51.361746 20:S ping packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:51.361756 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:51.361764 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:51.361771 20:S --- Processing packet of type ping, 2464 bytes
23:46:51.361961 20:S ping packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:51.361969 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:51.361976 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:51.363043 20:S --- Processing packet of type ping, 2568 bytes
23:46:51.363064 20:S ping packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:51.363071 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:51.363081 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:51.363087 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:51.366320 20:S Client closed connection id=164 addr=127.0.0.1:37364 laddr=127.0.0.1:6380 fd=15 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=slowlog|get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=252 tot-net-out=18641 tot-cmds=8
23:46:52.146326 20:S Pinging node dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:52.146735 20:S --- Processing packet of type pong, 2568 bytes
23:46:52.146749 20:S pong packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:52.146754 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:52.146758 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:52.146841 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:52.181045 20:S Accepted 192.168.59.124:48884
23:46:52.208373 20:S Client closed connection id=166 addr=192.168.59.124:48884 laddr=192.168.142.191:6380 fd=15 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=60 tot-net-out=1391 tot-cmds=2
23:46:52.209252 20:S Accepted 192.168.59.124:48894
23:46:52.246863 20:S DB 0: 28462 keys (20798 volatile) in 106340 slots HT.
23:46:52.246881 20:S Total: 18 clients connected (0 replicas), 143047248 (136.42M) bytes in use
23:46:52.246889 20:S Connecting to PRIMARY 127.0.0.1:6380
23:46:52.246940 20:S PRIMARY <-> REPLICA sync started
23:46:52.247014 20:S Accepted 127.0.0.1:38054
23:46:52.248831 20:S Non blocking connect for SYNC fired the event.
23:46:57.486288 20:S Failed to read response from the server: (null)
23:46:57.486334 20:S Primary did not respond to command during SYNC handshake
23:46:57.486936 20:S Client closed connection id=168 addr= laddr=127.0.0.1:6380 fd=22 name=*redacted* age=5 idle=5 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=ping user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=14 tot-net-out=7 tot-cmds=1
23:46:57.487108 20:S Client closed connection id=167 addr=192.168.59.124:48894 laddr=192.168.142.191:6380 fd=15 name=*redacted* age=5 idle=5 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=133 tot-net-out=1396 tot-cmds=3
23:46:57.487272 20:S Accepted 192.168.59.124:48910
23:46:57.488190 20:S Error accepting a client connection: error:0A000126:SSL routines::unexpected eof while reading (addr= laddr=192.168.142.191:6380)
23:46:57.488241 20:S --- Processing packet of type ping, 2464 bytes
23:46:57.488276 20:S ping packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:57.488282 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:57.488338 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:57.488345 20:S I/O error reading from node link: connection closed
23:46:57.488445 20:S --- Processing packet of type ping, 2360 bytes
23:46:57.488478 20:S ping packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:57.488486 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:57.488490 20:S I/O error reading from node link: connection closed
23:46:57.488577 20:S --- Processing packet of type ping, 2464 bytes
23:46:57.488610 20:S ping packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:57.488617 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:57.488622 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:57.488691 20:S I/O error reading from node link: connection closed
23:46:57.488750 20:S Client closed connection id=116 addr=192.168.175.183:48706 laddr=192.168.142.191:6380 fd=41 name=*redacted* age=235 idle=25 flags=tB db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22424 events=r cmd=ping user=*redacted* redir=0 resp=3 lib-name=rueidis lib-ver=1.0.62 tot-net-in=315 tot-net-out=201 tot-cmds=10
23:46:57.489028 20:S Accepting cluster node connection from 192.168.97.127:55250
23:46:57.489818 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:57.491537 20:S Accepted 192.168.59.94:57636
23:46:57.492379 20:S Client closed connection id=152 addr=127.0.0.1:36980 laddr=127.0.0.1:6380 fd=24 name=*redacted* age=12 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=457 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=cluster|slots user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=242 tot-net-out=1720 tot-cmds=7
23:46:57.492554 20:S Client closed connection id=165 addr=192.168.97.86:56090 laddr=192.168.142.191:6380 fd=16 name=*redacted* age=6 idle=0 flags=tB db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=7 obl=0 oll=0 omem=0 tot-mem=22424 events=r cmd=ping user=*redacted* redir=0 resp=3 lib-name=rueidis lib-ver=1.0.62 tot-net-in=274 tot-net-out=1477 tot-cmds=6
23:46:57.492725 20:S Accepting cluster node connection from 192.168.175.143:40770
23:46:57.493380 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:57.494648 20:S --- Processing packet of type pong, 2672 bytes
23:46:57.494668 20:S pong packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:57.494674 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:57.494678 20:S GOSSIP c6ebc92857bc76de852a131a006074746827e468 192.168.59.67:6380@16380 master
23:46:57.494797 20:S GOSSIP 880f5c242944d3fdd094fbb343e48ffc1ff8a997 192.168.142.160:6380@16380 master
23:46:57.494815 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:57.494822 20:S Error writing to client: Broken pipe
23:46:57.494914 20:S Error writing to client: Broken pipe
23:46:57.494960 20:S Accepted 192.168.142.179:36046
23:46:57.495885 20:S Accepting cluster node connection from 192.168.97.101:40924
23:46:57.496583 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:57.497533 20:S --- Processing packet of type pong, 2672 bytes
23:46:57.497549 20:S pong packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:57.497553 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:57.497632 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:57.497641 20:S GOSSIP 880f5c242944d3fdd094fbb343e48ffc1ff8a997 192.168.142.160:6380@16380 master
23:46:57.497645 20:S GOSSIP c6ebc92857bc76de852a131a006074746827e468 192.168.59.67:6380@16380 master
23:46:57.499463 20:S --- Processing packet of type pong, 2568 bytes
23:46:57.499478 20:S pong packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:57.499482 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:57.499560 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:57.499568 20:S GOSSIP 880f5c242944d3fdd094fbb343e48ffc1ff8a997 192.168.142.160:6380@16380 master
23:46:57.499614 20:S Accepted 192.168.59.124:48918
23:46:57.500340 20:S Error accepting a client connection: error:0A000126:SSL routines::unexpected eof while reading (addr=192.168.59.124:48918 laddr=192.168.142.191:6380)
23:46:57.500418 20:S Accepting cluster node connection from 192.168.142.160:42006
23:46:57.501159 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:57.503746 20:S Accepted 192.168.59.111:45050
23:46:57.504512 20:S Accepting cluster node connection from 192.168.59.67:52886
23:46:57.505247 20:S Error accepting cluster node connection: error:0A000126:SSL routines::unexpected eof while reading
23:46:57.505753 20:S Accepted 192.168.97.104:52596
23:46:57.506460 20:S Accepting cluster node connection from 192.168.97.127:55256
23:46:57.506925 20:S Accepting cluster node connection from 192.168.175.143:45214
23:46:57.507451 20:S Accepting cluster node connection from 192.168.97.101:40940
23:46:57.508000 20:S Accepting cluster node connection from 192.168.142.160:44510
23:46:57.508596 20:S Accepting cluster node connection from 192.168.59.67:45362
23:46:57.509077 20:S Accepted 127.0.0.1:38484
23:46:57.509566 20:S Error accepting a client connection: error:0A000126:SSL routines::unexpected eof while reading (addr= laddr=127.0.0.1:6380)
23:46:57.511824 20:S Accepted 192.168.59.124:48920
23:46:57.512321 20:S Error accepting a client connection: error:0A000126:SSL routines::unexpected eof while reading (addr=192.168.59.124:48920 laddr=192.168.142.191:6380)
23:46:57.517896 20:S Accepted 127.0.0.1:42364
23:46:57.518451 20:S Error accepting a client connection: error:0A000126:SSL routines::unexpected eof while reading (addr= laddr=127.0.0.1:6380)
23:46:57.518534 20:S --- Processing packet of type ping, 2672 bytes
23:46:57.518552 20:S ping packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:46:57.518560 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:46:57.518568 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:57.518574 20:S GOSSIP c6ebc92857bc76de852a131a006074746827e468 192.168.59.67:6380@16380 master
23:46:57.518866 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:57.518881 20:S --- Processing packet of type ping, 2616 bytes
23:46:57.518887 20:S Failover election in progress for epoch 17, but received a claim from node 880f5c242944d3fdd094fbb343e48ffc1ff8a997 () with an equal or higher epoch 19. Resetting the election since we cannot win an election in the past.
23:46:57.518896 20:S ping packet received: 880f5c242944d3fdd094fbb343e48ffc1ff8a997
23:46:57.518904 20:S node 880f5c242944d3fdd094fbb343e48ffc1ff8a997 () announces that it is a primary in shard 3bef3dc2d6c1d29eba0207f7608fc51af4681cf1
23:46:57.518917 20:S GOSSIP c6ebc92857bc76de852a131a006074746827e468 192.168.59.67:6380@16380 master
23:46:57.518925 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:57.518935 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:57.518947 20:S --- Processing packet of type ping, 2616 bytes
23:46:57.518953 20:S ping packet received: c6ebc92857bc76de852a131a006074746827e468
23:46:57.518964 20:S node c6ebc92857bc76de852a131a006074746827e468 () announces that it is a primary in shard b29293d5f0dc9389cefeb1e71f71999836b4ccea
23:46:57.518973 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:57.518983 20:S GOSSIP 880f5c242944d3fdd094fbb343e48ffc1ff8a997 192.168.142.160:6380@16380 master
23:46:57.518990 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:57.518995 20:S --- Processing packet of type ping, 2672 bytes
23:46:57.519007 20:S ping packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:57.519014 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:57.519020 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:57.519030 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:57.519037 20:S GOSSIP 880f5c242944d3fdd094fbb343e48ffc1ff8a997 192.168.142.160:6380@16380 master
23:46:57.519046 20:S --- Processing packet of type ping, 2568 bytes
23:46:57.519053 20:S ping packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:46:57.519063 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:57.519070 20:S GOSSIP c6ebc92857bc76de852a131a006074746827e468 192.168.59.67:6380@16380 master
23:46:57.519078 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:57.521377 20:S Accepted 192.168.97.86:41224
23:46:57.522262 20:S Accepted 192.168.97.86:41238
23:46:57.522771 20:S Accepted 192.168.59.124:33156
23:46:57.523303 20:S Error accepting a client connection: error:0A000126:SSL routines::unexpected eof while reading (addr=192.168.59.124:33156 laddr=192.168.142.191:6380)
23:46:57.523370 20:S Accepted 127.0.0.1:42462
23:46:57.593489 20:S Connecting with Node 880f5c242944d3fdd094fbb343e48ffc1ff8a997 at 192.168.142.160:16380
23:46:57.595388 20:S Connecting with Node c6ebc92857bc76de852a131a006074746827e468 at 192.168.59.67:16380
23:46:57.595561 20:S --- Processing packet of type pong, 2616 bytes
23:46:57.595569 20:S pong packet received: 880f5c242944d3fdd094fbb343e48ffc1ff8a997
23:46:57.595574 20:S node 880f5c242944d3fdd094fbb343e48ffc1ff8a997 () announces that it is a primary in shard bc2be14c2318f6790a9fe88efd4ecb7dc37b87f1
23:46:57.595645 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:57.595648 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:57.595652 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:57.595714 20:S I/O error reading from node link: connection closed
23:46:57.597311 20:S I/O error reading from node link: connection closed
23:46:57.597902 20:S --- Processing packet of type pong, 2616 bytes
23:46:57.597911 20:S pong packet received: c6ebc92857bc76de852a131a006074746827e468
23:46:57.597916 20:S node c6ebc92857bc76de852a131a006074746827e468 () announces that it is a primary in shard ccaf39b98971df85eba3e2c6dc90215b8ba5b6ee
23:46:57.597987 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:46:57.597991 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:46:57.597995 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:57.601331 20:S Accepted 192.168.59.124:33172
23:46:57.696708 20:S Client closed connection id=182 addr=192.168.59.124:33172 laddr=192.168.142.191:6380 fd=22 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=133 tot-net-out=1606 tot-cmds=3
23:46:57.711562 20:S Accepted 192.168.59.124:33174
23:46:57.782495 20:S Client closed connection id=183 addr=192.168.59.124:33174 laddr=192.168.142.191:6380 fd=22 name=*redacted* age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=133 tot-net-out=1606 tot-cmds=3
23:46:57.787248 20:S Accepted 192.168.59.124:33186
23:46:58.180979 20:S Client closed connection id=184 addr=192.168.59.124:33186 laddr=192.168.142.191:6380 fd=22 name=*redacted* age=1 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=0 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=info user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=133 tot-net-out=1606 tot-cmds=3
23:46:58.294940 20:S Pinging node dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:58.337100 20:S --- Processing packet of type pong, 2672 bytes
23:46:58.337128 20:S pong packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:46:58.337138 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:46:58.337145 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:46:58.337160 20:S GOSSIP c6ebc92857bc76de852a131a006074746827e468 192.168.59.67:6380@16380 master
23:46:58.337167 20:S GOSSIP 880f5c242944d3fdd094fbb343e48ffc1ff8a997 192.168.142.160:6380@16380 master
23:46:58.394686 20:S Accepted 192.168.59.124:33194
23:46:58.395950 20:S Connecting to PRIMARY 127.0.0.1:6380
23:46:58.396005 20:S PRIMARY <-> REPLICA sync started
23:46:58.396034 20:S Accepted 127.0.0.1:42628
23:46:58.399398 20:S Non blocking connect for SYNC fired the event.
23:47:03.630225 20:S Failed to read response from the server: (null)
23:47:03.630258 20:S Primary did not respond to command during SYNC handshake
23:47:03.630682 20:S Client closed connection id=186 addr= laddr=127.0.0.1:6380 fd=41 name=*redacted* age=5 idle=5 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=37760 events=r cmd=ping user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=14 tot-net-out=7 tot-cmds=1
23:47:03.630846 20:S --- Processing packet of type ping, 2672 bytes
23:47:03.630908 20:S ping packet received: dc81be9ba4688a18d123dca363b94ba4815e0e01
23:47:03.630924 20:S node dc81be9ba4688a18d123dca363b94ba4815e0e01 () announces that it is a primary in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:47:03.630932 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:47:03.630939 20:S GOSSIP 880f5c242944d3fdd094fbb343e48ffc1ff8a997 192.168.142.160:6380@16380 master
23:47:03.630946 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:47:03.631129 20:S I/O error reading from node link: connection closed
23:47:03.631140 20:S --- Processing packet of type ping, 2672 bytes
23:47:03.631144 20:S ping packet received: ff19d2219d76a6d8959424b4de4a025b1f3632e6
23:47:03.631149 20:S node ff19d2219d76a6d8959424b4de4a025b1f3632e6 () announces that it is a replica in shard ece4ebf764d0ed5628948b823e93b5f1521f657d
23:47:03.631152 20:S GOSSIP c6ebc92857bc76de852a131a006074746827e468 192.168.59.67:6380@16380 master
23:47:03.631158 20:S GOSSIP 03124dd2ecbc25c4b96af5b29ae56d234b865c86 192.168.97.101:6380@16380 master
23:47:03.631163 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:47:03.631175 20:S I/O error reading from node link: connection closed
23:47:03.631181 20:S --- Processing packet of type ping, 2672 bytes
23:47:03.631418 20:S ping packet received: 03124dd2ecbc25c4b96af5b29ae56d234b865c86
23:47:03.631429 20:S node 03124dd2ecbc25c4b96af5b29ae56d234b865c86 () announces that it is a primary in shard a1a9e5f58cddbb440fdbeca348506b1fa2728f1a
23:47:03.631433 20:S GOSSIP dc81be9ba4688a18d123dca363b94ba4815e0e01 192.168.175.143:6380@16380 master
23:47:03.631437 20:S GOSSIP ff19d2219d76a6d8959424b4de4a025b1f3632e6 192.168.97.127:6380@16380 slave
23:47:03.631440 20:S GOSSIP 880f5c242944d3fdd094fbb343e48ffc1ff8a997 192.168.142.160:6380@16380 master
...