Consumer filter subjects stop matching after leader step-down

[dnie]

Observed behavior

Consumer stops working after filterSubject has been modified (edit) and the consumer leader has been stepped down.
Also all messages are dropped even if they match the filter subjects.

• Unexpected: Unprocessed Messages on the consumer remains at 0
• Unexpected: Stream message count also no longer increasing (messages are lost)
• Unexpected: nats consumer next returns: nats: error: no message received: nats: timeout

Expected behavior

• Consumer should match published messages against filter subjects
• Unprocessed Messages should increase when matching messages are published
• nats consumer next should return the published messages after step-down

Server and client version

NATS server 2.12.3
NATS client 0.3.0
NATS GO SDK v1.48.0

Host environment

5 Node NATS Cluster running on AWS EKS
(also on a local K3s cluster)

Steps to reproduce

1. Create a stream and consumer

# other params are default
nats stream add foo-stream  --subjects "foo.>" --replicas 5 --storage file --retention interest --ack --discard old --dupe-window 2m --no-allow-direct

# other params are default
nats consumer add foo-stream foo-consumer --pull --filter "foo.bar" --filter "foo.bar1" --deliver all  --ack explicit    --replay instant   --max-deliver 20   --max-pending 1000

2. Check everything

# use 2nd terminal to put some data in
nats pub "foo.bar" 'check {{ .Count }}' --sleep 1s --count 5

# use 3rd terminal to get messages from the consumer
nats consumer next foo-stream foo-consumer --wait 2s --count 5

# use 4th terminal to monitor the consumer state itself
watch nats consumer info foo-stream foo-consumer

3. Now trigger the issue by editing the consumer

nats consumer edit foo-stream foo-consumer  --filter "foo.bar" --filter "foo.bar2"

# ==> right here Step 2 has expected behavior

nats consumer cluster step-down foo-stream foo-consumer

# ==> staring here Step 2 has unexpected behavior

[MauriceVanVeen]

Fixed in #7773