Skip to content

Messages from a deleted consumer are not deleted as expected in interest based stream #7754

@superlevure

Description

@superlevure

Observed behavior

We have a stream called stream-a where a consumer called consumer-a was deleted as expected after its inactive threshold expired, but we suspect its unprocessed messages were not as we were expecting.

The stream shows 142,121 messages, of which 19,680 can be traced back to consumers via nats consumer ls stream-a but 122,441 can't.

When performing a nats subjects stream-a we find back the 122,441 messages and they are tied to the subject used by the deleted consumer (see the result of each command below).

nats stream info stream-a:

             Description: XXX
             Subjects: a.b.c.d.e.f.*.h.*.*
             Replicas: 3
              Storage: File

Options:

               Retention: Interest
         Acknowledgments: true
          Discard Policy: Old
        Duplicate Window: 2m0s
    Allows Batch Publish: false
         Allows Counters: false
       Allows Msg Delete: true
  Allows Per-Message TTL: false
            Allows Purge: true
        Allows Schedules: false
          Allows Rollups: false

Limits:

        Maximum Messages: unlimited
     Maximum Per Subject: unlimited
           Maximum Bytes: unlimited
             Maximum Age: 14d0h0m0s
    Maximum Message Size: unlimited
       Maximum Consumers: unlimited

Cluster Information:

                    Name: nats
           Cluster Group: S-R3F-6nkh1i6A
                  Leader: nats-1 (3d0h20m40s)
                 Replica: nats-0, current, seen 675ms ago
                 Replica: nats-2, current, seen 675ms ago

State:

            Host Version: 2.12.3
      Required API Level: 0 hosted at level 2
                Messages: 142,121
                   Bytes: 183 MiB
          First Sequence: 243,859 @ 2026-01-20 05:47:49
           Last Sequence: 1,292,322 @ 2026-01-22 15:54:23
        Deleted Messages: 906,343
        Active Consumers: 12
      Number of Subjects: 2

nats consumer info stream-a consumer-a (this was run few hours before the consumer was deleted by the server)

Configuration:

                    Name: consumer-a
               Pull Mode: true
          Filter Subject: a.b.c.d.e.f.g.h.i.j
          Deliver Policy: All
              Ack Policy: Explicit
                Ack Wait: 30.00s
           Replay Policy: Instant
         Max Ack Pending: 1,000
       Max Waiting Pulls: 512
      Inactive Threshold: 1d0h0m0s

Cluster Information:

                    Name: nats
              Raft Group: C-R3F-ktq1G6zJ
                  Leader: nats-1 (21h18m33s)
                 Replica: nats-0, current, seen 389ms ago
                 Replica: nats-2, current, seen 389ms ago

State:

            Host Version: 2.12.3
      Required API Level: 0 hosted at level 2
  Last Delivered Message: Consumer sequence: 57,278 Stream sequence: 240,582 Last delivery: 9h0m51s ago
    Acknowledgment Floor: Consumer sequence: 57,278 Stream sequence: 240,582 Last Ack: 9h0m51s ago
        Outstanding Acks: 0 out of maximum 1,000
    Redelivered Messages: 0
    Unprocessed Messages: 52,431
           Waiting Pulls: 0 of maximum 512

nats stream subjects stream-a

{
  "<another_subject>": 19680,
  "a.b.c.d.e.f.g.h.i.j": 122441
}

nats server stream-check --stream stream-a shows the stream is in sync:

Stream Replica,Raft,Account,Account ID,Node,Messages,Bytes,Subjects,Deleted,Consumers,First,Last,Status,Leader,Peers,
stream-a,S-R3F-6nkh1i6A,default,default,nats-0,142121,191421381,2,920023,12,243859,1306002,IN SYNC,nats-1,"nats-2(current=false\,offline=false)      nats-1(current=true \,offline=false)",
stream-a,S-R3F-6nkh1i6A,default,default,nats-1*,142121,191421381,2,920023,12,243859,1306002,IN SYNC,nats-1,"nats-2(current=true \,offline=false)      nats-0(current=true \,offline=false)",
stream-a,S-R3F-6nkh1i6A,default,default,nats-2,142121,191421381,2,920023,12,243859,1306002,IN SYNC,nats-1,"nats-1(current=true \,offline=false)      nats-0(current=false\,offline=false)",

Thank you for your help 🙏

Expected behavior

The stream should delete the messages associated with a deleted consumer

Server and client version

  • server version: 2.12.3

Host environment

Kubernetes based deployment, using the official helm chart.

Steps to reproduce

This is the first time it happens to us so we can't reproduce consistently, but the steps were:

  • Create stream-a
  • Create consumer-a
  • Publish messages to consumer-a subject while it's offline
  • Wait for the server to delete consumer-a after it expires
    -> the server don't delete the messages associated with consumer-a

Metadata

Metadata

Labels

defectSuspected defect such as a bug or regression

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions