Built-In Redundancy Monitoring for GoAlert Instances

[mastercactapus]

What problem would you like to solve? Please describe:
There is a need for a feature to facilitate the monitoring of a GoAlert production instance employing distinct installations of GoAlert. While certain elements of this process are partially implemented, they are not documented, complex, and possibly outdated.

Describe the solution you'd like:
A new admin page for Remote Monitor to streamline the configuration and operation of these monitoring services. Features required:

• Auto-creation of a new monitor service with a webhook, pointing back to the main instance.
• Auto-sync or manual sync options for updating the monitor environments from the primary GoAlert instance.
• Enable the syncing of on-call users from the primary instance to the monitor installations. Consider a solution for user contact method validation since notifications may come from a new number.

Describe alternatives you've considered:
The current procedure involves a command, 'monitor', and the 'remotemonitor' package, to create and manage added complexity redundancy measures.

Additional context:
This feature request caters primarily to the admins responsible for a GoAlert installation rather than the application's end users.

[mastercactapus]

A good experience will rely heavily on #3007

[mastercactapus]

Current state reference info

Current check sequence

Current Check Deployment Example

---

Worth noting that the current operation requires remote monitor to be deployed with a config file with various required settings, it's own twilio number, and be publicly-routable

[mastercactapus]

For next steps:

MVP: minimal API/db additions to allow create alert -> heartbeat using webhook on an EP

+1: auto sync on-call users
+1: auto config from production/main instance

+2: "sentinel mode" -- slimmer UI, warning banner, make it obvious an instance is for monitoring another (with link)
+3: first time setup /w sentinel

Separate supporting feature ideas:

• admin-only resources (i.e., don't allow regular users to change, or possibly see, a monitoring/synced service)
• app config to set the default theme color (help differentiate separate envs)
• Add Context and Information to Heartbeat Monitors and Integration Keys #3024

---

Implementation thoughts/notes:

• have sentinels do as much work as possible (PULL config rather than push for on-call)
• synthetic check: can I create an alert? did I get a notification? -> POST heartbeat(s)