Keeping silences in memory and not writing to disk until exit.

[vegetablesalad]

I'm setting up my alertmanager container and deploying it.
While doing this I discovered a weird issue where I'm having problem persisting silence data.
I spent a lot of time trying to find where the date is stored as the default location never had anything, and the --storage.path= location was always empty as well.

So after a while I discovered that data is being written only when I shutdown the alertmanager, and it has to be a graceful shutdown. Only after that I see something in the data folder.

I expected for silence data to be written as soon as a new Silence is added.

This is somewhat messing with persisting data between deployments, as not all the container shutdowns and recreations are triggering database creation.

Linux 3.10.0-1062.12.1.el7.x86_64 x86_64 - the same on my macbook as well.

alertmanager, version 0.22.2 (branch: HEAD, revision: 44f8adc06af5101ad64bd8b9c8b18273f2922051)
  build user:       root@b595c7f32520
  build date:       20210602-07:50:37
  go version:       go1.16.4
  platform:         linux/amd64

[gotjosh]

By default, silences are persisted to disk every 15 minutes or on a clean shutdown as you just mentioned.

This is somewhat messing with persisting data between deployments

Can you share more details on what's going on here? It seems like this is particular with your setup and not necessarily an Alertmanager problem.

[gotjosh]

Writing data on disk as soon as it's added would cause write amplification and thus not desirable for a wider audience.

As of 0.25 we've included a new flag to control how often you want to flush to disk. In your setup, you're able to set this to as long as you want e.g. 5s

See: #2849