Task: Improvements to silences

[grobinson-grafana]

There are a number of issues with how silences work that I plan to fix:

1. silences.Set(*pb.Silence) returns (id, error), where error is non-nil on failure. However, in some cases silences.Set will set the ID for *pb.Silence and return it even when the silence could not be stored (i.e. because it failed validation or would exceed limits). This ID does not exist, and does not refer to an actual stored silence. Subsequent calls to silences.Set will return ErrNotFound.

   • Remove Id return from silences.Set(*pb.Silence) #3881

2. When an update to a silence requires expiring and re-creating the silence (i.e. when canUpdate returns false), if the new silence cannot be stored (i.e. because it failed validation or would exceed limits) the old silence is still expired. Updates should be atomic such that partial updates are not possible.

   • Fix MaxSilences limit causes incomplete updates of existing silences #3877
   • Fix MaxSilenceSizeBytes limit causes incomplete updates of existing silences #3897

3. The validation of silences happens in silences.setSilence, and this creates a lot of odd cases in the code. For example, we have a boolean skipValidate that is used by expire to ensure existing silences can still be expired if the validation rules or limits change after the silence was created. It should be moved elsewhere.

   • Fix MaxSilenceSizeBytes limit causes incomplete updates of existing silences #3897

4. Like other fields, I think the maintenance function should be initialized in silences.New. It should be possible to replace the maintenance function at runtime with silences.SetMaintenanceFunc.