add prefect server services CLI commands

[zzstoatzz]

wondering if maybe should be experimental as I think we may want to add some logging config and maybe update the signature of things - feedback?

• start: start all loop services in the foreground
  • --background start them in the background
• ls: list all known loop services and whether they are enabled
• stop: stop any running loop services

both start and stop should be idempotent

in order to ensure the --background option can be managed by a single process, I created a hidden manager command under services (which does not show up in --help). open to other suggestions

demo usage

» prefect server services start

Starting services... Press CTRL+C to stop

23:16:55.600 | INFO    | prefect.cli.server - Started service: CancellationCleanup
23:16:55.601 | INFO    | prefect.cli.server - Started service: FlowRunNotifications
23:16:55.601 | INFO    | prefect.cli.server - Started service: Foreman
23:16:55.601 | INFO    | prefect.cli.server - Started service: MarkLateRuns
23:16:55.601 | INFO    | prefect.cli.server - Started service: FailExpiredPauses
23:16:55.601 | INFO    | prefect.cli.server - Started service: RecentDeploymentsScheduler
23:16:55.602 | INFO    | prefect.cli.server - Started service: Scheduler
23:16:55.602 | INFO    | prefect.cli.server - Started service: Telemetry
23:16:55.602 | INFO    | prefect.cli.server - Started service: ProactiveTriggers
^C23:16:56.784 | INFO    | prefect.cli.server - Received cancellation, stopping services...
23:16:56.784 | INFO    | prefect.cli.server - Stopped service: CancellationCleanup
23:16:56.784 | INFO    | prefect.cli.server - Stopped service: FlowRunNotifications
23:16:56.785 | INFO    | prefect.cli.server - Stopped service: Foreman
23:16:56.785 | INFO    | prefect.cli.server - Stopped service: MarkLateRuns
23:16:56.785 | INFO    | prefect.cli.server - Stopped service: FailExpiredPauses
23:16:56.785 | INFO    | prefect.cli.server - Stopped service: RecentDeploymentsScheduler
23:16:56.785 | INFO    | prefect.cli.server - Stopped service: Scheduler
23:16:56.785 | INFO    | prefect.cli.server - Stopped service: Telemetry
23:16:56.786 | INFO    | prefect.cli.server - Stopped service: ProactiveTriggers

All services stopped.
» prefect server services start --background
Starting service: CancellationCleanup
Starting service: FlowRunNotifications
Starting service: Foreman
Starting service: MarkLateRuns
Starting service: FailExpiredPauses
Starting service: RecentDeploymentsScheduler
Starting service: Scheduler
Starting service: Telemetry
Starting service: ProactiveTriggers

Services are running in the background.
Use `prefect server services stop` to stop them.
» prefect server services stop

Shutting down...
✓ Services stopped

All services stopped.
» prefect server services ls
                                                                  Available Services
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                         ┃ Enabled?  ┃ Description                                                                                             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ CancellationCleanup          │ True      │ A simple loop service responsible for cancelling tasks and subflows left over from                      │
│ FlowRunNotifications         │ True      │ A loop service that checks for flow run notifications that need to be sent.                             │
│ Foreman                      │ True      │ A loop service responsible for monitoring the status of workers.                                        │
│ MarkLateRuns                 │ True      │ A simple loop service responsible for identifying flow runs that are "late".                            │
│ FailExpiredPauses            │ True      │ A simple loop service responsible for identifying Paused flow runs that no longer can be resumed.       │
│ RecentDeploymentsScheduler   │ True      │ A scheduler that only schedules deployments that were updated very recently.                            │
│ Scheduler                    │ True      │ A loop service that schedules flow runs from deployments.                                               │
│ Telemetry                    │ True      │ This service sends anonymous data (e.g. count of flow runs) to Prefect to help us                       │
│ ProactiveTriggers            │ True      │ A loop service that runs the proactive triggers consumer                                                │
└──────────────────────────────┴───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────┘

[codspeed-hq]

CodSpeed Performance Report

Merging #16706 will not alter performance

_{Comparing services-start (00c2373) with main (989321a)}

Summary

✅ 2 untouched benchmarks

[cicdw]

a few small comments, overall the implementation makes sense to me - I'll probably give it another pass in a bit

[cicdw]

two minor points:

• given this is an important file and a global, I suggest initializing it (but not creating it) at the top of the file and calling it PID_FILE
• to mirror the server start command, I think it might be more consistent to use Path(PREFECT_HOME.value()) / "services.pid" rather than creating a new subdirectory

[zzstoatzz]

good calls @cicdw! suggestions applied in 0bbd572

[cicdw]

Propose to downgrade many of the new logs to DEBUG: I think anything that displays "implementation" information like service names is only useful when trying to understand what's going on under the hood, hence DEBUG vs. INFO which I think is broad strokes useful information relevant to all users ("services all started", etc.).

[cicdw]

LGTM