Make health probe server more general purpose

This removes the health check logic from the ProbeServer and renames the
ProbeServer to UtilityServer that accepts any Rack based app.

The health check and catchall logic are moved into simple Rack middleware
that can be composed by users however they like and be used to preserve
existing health check behavior while transitioning to a more general
purpose utility server.

All and all this pattern will allow users to add whatever functionality
they like to GoodJob's web server by composing Rack apps and using
GoodJob's configuration to pass in users' Rack apps. IE:

```
config.good_job.middleware = Rack::Builder.app do
  use GoodJob::Middleware::MyCustomMiddleware
  use GoodJob::Middleware::PrometheusExporter
  use GoodJob::Middleware::Healthcheck
  run GoodJob::Middleware::CatchAll
end
config.good_job.middleware_port = 7001
```

This could help resolve:

* bensheldon#750
* bensheldon#532

lib/good_job.rb

@@ -30,8 +30,10 @@
 require "good_job/multi_scheduler"
 require "good_job/notifier"
 require "good_job/poller"
+require "good_job/middleware/catch_all"
+require "good_job/middleware/healthcheck"
 require "good_job/http_server"
-require "good_job/probe_server"
+require "good_job/utility_server"
 require "good_job/scheduler"
 require "good_job/shared_executor"
 require "good_job/systemd_service"

lib/good_job/cli.rb

@@ -101,8 +101,10 @@ def start
       capsule.start
       systemd.start
 
-      if configuration.probe_port
-        probe_server = GoodJob::ProbeServer.new(port: configuration.probe_port)
+      middleware = Rails.application.config.good_job.middleware
+      port = Rails.application.config.good_job.middleware_port
+      if middleware && port
+        probe_server = GoodJob::UtilityServer.new(app: middleware, port: port)
         probe_server.start
       end
 

lib/good_job/middleware/catch_all.rb

@@ -0,0 +1,9 @@
+module GoodJob
+  module Middleware
+    class CatchAll
+      def self.call(env)
+        [404, {}, ["Not found"]]
+      end
+    end
+  end
+end

lib/good_job/middleware/healthcheck.rb

@@ -0,0 +1,25 @@
+module GoodJob
+  module Middleware
+    class Healthcheck
+      def initialize(app)
+        @app = app
+      end
+
+      def call(env)
+        case Rack::Request.new(env).path
+        when '/', '/status'
+          [200, {}, ["OK"]]
+        when '/status/started'
+          started = GoodJob::Scheduler.instances.any? && GoodJob::Scheduler.instances.all?(&:running?)
+          started ? [200, {}, ["Started"]] : [503, {}, ["Not started"]]
+        when '/status/connected'
+          connected = GoodJob::Scheduler.instances.any? && GoodJob::Scheduler.instances.all?(&:running?) &&
+                      GoodJob::Notifier.instances.any? && GoodJob::Notifier.instances.all?(&:listening?)
+          connected ? [200, {}, ["Connected"]] : [503, {}, ["Not connected"]]
+        else
+          @app.call(env)
+        end
+      end
+    end
+  end
+end

lib/good_job/utility_server.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+module GoodJob
+  class UtilityServer
+    def self.task_observer(time, output, thread_error) # rubocop:disable Lint/UnusedMethodArgument
+      return if thread_error.is_a? Concurrent::CancelledOperationError
+
+      GoodJob._on_thread_error(thread_error) if thread_error
+    end
+
+    def initialize(app:, port:)
+      @port = port
+      @app = app
+    end
+
+    def start
+      @handler = HttpServer.new(@app, port: @port, logger: GoodJob.logger)
+      @future = Concurrent::Future.new { @handler.run }
+      @future.add_observer(self.class, :task_observer)
+      @future.execute
+    end
+
+    def running?
+      @handler&.running?
+    end
+
+    def stop
+      @handler&.stop
+      @future&.value # wait for Future to exit
+    end
+  end
+end