patch default user notification policy changes + fix failing e2e test

[joeyorlando]

What this PR does

This is a follow-up PR to #4628. As @Ferril pointed out, there was a slight issue in apps.alerts.tasks.notify_user.perform_notification method when using a "fallback"/default user notification policy. This is because the log_record_pk arg passed into perform_notification will fetch the UserNotificationPolicyLogRecord object, but that object will have a notification_policy set to None (because there's no persistent UserNotificationPolicy object to refer to).

Instead we now pass in a second argument to perform_notification, use_default_notification_policy_fallback. If this is true, simply grab the transient/in-memory UserNotificationPolicy and use that inside of this task

Related to #4410

Checklist

• Unit, integration, and e2e (if applicable) tests updated
• Documentation added (or pr:no public docs PR label added if not required)
• Added the relevant release notes label (see labels prefixed w/ release:). These labels dictate how your PR will
  show up in the autogenerated release notes.

[joeyorlando]

These changes should fix this failing e2e test (which started failing after we recently got rid of autogenerating default user notification policy steps)

[Ferril]

I found another issue. It seems like there is an endless loop of notification. In the notify_user_task we call next notify_user_task here (line 231):

transaction.on_commit(
    partial(
        notify_user_task.apply_async,
        (user.pk, alert_group.pk, notification_policy.pk, reason),
        ...
  )

The next task will start with previous_notification_policy_pk=None, notify user again and call next notify_user_task with the same parameters -> 🔁

The regular flow is:

1. call notify_user_task with previous_notification_policy_pk=None
2. get the first notification policy
3. create log with type TRIGGERED and call perform_notification
4. call next notify_user_task with previous_notification_policy_pk=<current notification_policy.pk>
5. in the next notify_user_task get notifiction_policy.next()
6. if it's not None: repeat from step 3. Otherwise create notification log with type FINISHED and stop notification.

So for one existing notification step the notify_user_task is called twice.
For default notification we can add a flag to notify_user_task when we call it again OR we can skip steps 4 and 5, create FINISHED log and stop notification 🤔 wdyt?

[joeyorlando]

great catch and thanks for the really detailed breakdown, this was super helpful in understanding the flow of things 🫶 This should now be addressed in 716ad63

Before

After

[Ferril]

LGTM 👍

[joeyorlando]

log_message is only ever updated here, but then it is never referenced/used anywhere