-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Description
Observed behavior
We have a stream called stream-a where a consumer called consumer-a was deleted as expected after its inactive threshold expired, but we suspect its unprocessed messages were not as we were expecting.
The stream shows 142,121 messages, of which 19,680 can be traced back to consumers via nats consumer ls stream-a but 122,441 can't.
When performing a nats subjects stream-a we find back the 122,441 messages and they are tied to the subject used by the deleted consumer (see the result of each command below).
nats stream info stream-a:
Description: XXX
Subjects: a.b.c.d.e.f.*.h.*.*
Replicas: 3
Storage: File
Options:
Retention: Interest
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 2m0s
Allows Batch Publish: false
Allows Counters: false
Allows Msg Delete: true
Allows Per-Message TTL: false
Allows Purge: true
Allows Schedules: false
Allows Rollups: false
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: 14d0h0m0s
Maximum Message Size: unlimited
Maximum Consumers: unlimited
Cluster Information:
Name: nats
Cluster Group: S-R3F-6nkh1i6A
Leader: nats-1 (3d0h20m40s)
Replica: nats-0, current, seen 675ms ago
Replica: nats-2, current, seen 675ms ago
State:
Host Version: 2.12.3
Required API Level: 0 hosted at level 2
Messages: 142,121
Bytes: 183 MiB
First Sequence: 243,859 @ 2026-01-20 05:47:49
Last Sequence: 1,292,322 @ 2026-01-22 15:54:23
Deleted Messages: 906,343
Active Consumers: 12
Number of Subjects: 2
nats consumer info stream-a consumer-a (this was run few hours before the consumer was deleted by the server)
Configuration:
Name: consumer-a
Pull Mode: true
Filter Subject: a.b.c.d.e.f.g.h.i.j
Deliver Policy: All
Ack Policy: Explicit
Ack Wait: 30.00s
Replay Policy: Instant
Max Ack Pending: 1,000
Max Waiting Pulls: 512
Inactive Threshold: 1d0h0m0s
Cluster Information:
Name: nats
Raft Group: C-R3F-ktq1G6zJ
Leader: nats-1 (21h18m33s)
Replica: nats-0, current, seen 389ms ago
Replica: nats-2, current, seen 389ms ago
State:
Host Version: 2.12.3
Required API Level: 0 hosted at level 2
Last Delivered Message: Consumer sequence: 57,278 Stream sequence: 240,582 Last delivery: 9h0m51s ago
Acknowledgment Floor: Consumer sequence: 57,278 Stream sequence: 240,582 Last Ack: 9h0m51s ago
Outstanding Acks: 0 out of maximum 1,000
Redelivered Messages: 0
Unprocessed Messages: 52,431
Waiting Pulls: 0 of maximum 512
nats stream subjects stream-a
{
"<another_subject>": 19680,
"a.b.c.d.e.f.g.h.i.j": 122441
}nats server stream-check --stream stream-a shows the stream is in sync:
Stream Replica,Raft,Account,Account ID,Node,Messages,Bytes,Subjects,Deleted,Consumers,First,Last,Status,Leader,Peers,
stream-a,S-R3F-6nkh1i6A,default,default,nats-0,142121,191421381,2,920023,12,243859,1306002,IN SYNC,nats-1,"nats-2(current=false\,offline=false) nats-1(current=true \,offline=false)",
stream-a,S-R3F-6nkh1i6A,default,default,nats-1*,142121,191421381,2,920023,12,243859,1306002,IN SYNC,nats-1,"nats-2(current=true \,offline=false) nats-0(current=true \,offline=false)",
stream-a,S-R3F-6nkh1i6A,default,default,nats-2,142121,191421381,2,920023,12,243859,1306002,IN SYNC,nats-1,"nats-1(current=true \,offline=false) nats-0(current=false\,offline=false)",
Thank you for your help 🙏
Expected behavior
The stream should delete the messages associated with a deleted consumer
Server and client version
- server version:
2.12.3
Host environment
Kubernetes based deployment, using the official helm chart.
Steps to reproduce
This is the first time it happens to us so we can't reproduce consistently, but the steps were:
- Create
stream-a - Create
consumer-a - Publish messages to
consumer-asubject while it's offline - Wait for the server to delete
consumer-aafter it expires
-> the server don't delete the messages associated withconsumer-a