NATS message bus has no working retention configured

[wkloucek]

Describe the bug

The NATS stream "main-queue" acts as single message bus for oCIS.

I expect the message bus to be ephemeral in the sense of:

• all delivery groups that subscribed get the messages
• all messages that are acknowledged are deleted

But that seems not to be the case.

Example of a oCIS instance running for a longer time:

~ # nats stream info main-queue
Information for Stream main-queue created 2023-07-24 13:53:49

             Subjects: main-queue
             Replicas: 1
              Storage: File

Options:

            Retention: Limits
     Acknowledgements: true
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false

Limits:

     Maximum Messages: unlimited
  Maximum Per Subject: unlimited
        Maximum Bytes: unlimited
          Maximum Age: unlimited
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited


Cluster Information:

                 Name: nats
               Leader: nats-1

State:

             Messages: 69,358
                Bytes: 66 MiB
             FirstSeq: 1 @ 2023-06-14T10:30:03 UTC
              LastSeq: 69,358 @ 2023-09-07T06:04:45 UTC
     Active Consumers: 10
   Number of Subjects: 1

You can see the main queue holding 69k messages.

This is because of the retention policy "limits".

When you consult https://docs.nats.io/using-nats/developer/develop_jetstream/model_deep_dive, you'll learn that limits means that retention happens when a limit is reached.

But you'll see that the oCIS default configuration does not set any limits on the main-queue stream.

There are two solutions:

• set limits to the stream (but which?)
• change the retention period (but which?)

Steps to reproduce

Steps to reproduce the behavior:

1. run oCIS with NATS (always the case)
2. use oCIS
3. see the main-queue growing forever

Expected behavior

main-queue does not grow forever but also deletes old messages

Actual behavior

main-queue grows forever

[wkloucek]

will be supersed if #7272 is implemented

[wkloucek]

KV store also seems to have no TTL:

~ # nats kv info eventhistory
Information for Key-Value Store Bucket eventhistory created 2024-01-23T09:55:17Z

Configuration:

          Bucket Name: eventhistory
         History Kept: 1
        Values Stored: 21
   Backing Store Kind: JetStream
          Description: KeyValue storage administered by go-micro store plugin
          Bucket Size: 32 KiB
  Maximum Bucket Size: unlimited
   Maximum Value Size: unlimited
          Maximum Age: unlimited
     JetStream Stream: KV_eventhistory
              Storage: File

Cluster Information:

                 Name: nats
               Leader: nats-2

[wkloucek]

main-queue: Maximum Age -> 14
eventhistory: Maximum Age -> 14d ??? (same as EVENTHISTORY_STORE_TTL)
caches: eg. storage-users -> Maximum Age -> 1d

[kobergj]

Did we check this with latest? I checked and all caches should now have ttls. Also eventhistory and userlog are hard set to 14 days.

main-queue max age is difficult to implement. Needs roundtrip through go-micro probably.

[wkloucek]

Did we check this with latest?

No with rc.2 because latest didn't start for some reason I can't recap

[micbar]

TTL should be implemented for all caches too now.