Node should panic in PoST supervised mode if service doesn't connect

[fasmat]

Description

There are a few ways in which a node can be configured to start in PoST supervised mode but fail to start smeshing. The most common ones are:

• grpc-private-services/grpc-tls-services are missing post handler and therefor do not accept connections from the post service
• node-address is not the correct address, i.e. it doesn't match the host + port defined by grpc-private-listener/grpc-tls-listener

This will result in the post service starting up and trying to connect every 5 seconds, but failing to do so.

With #5259 a possibly incorrect config is detected and logged by the node during startup. Additionally if the configuration causes the PoST service to be unable to connect to the node, the node should panic and shut down so a misconfigured setup doesn't go unnoticed by the user.

Acceptance criteria

• Configurable retries in post-service post-rs#151
• update the node supervisor to not try to restart the post service when it exits unexpectedly but instead panic with a message that the node is shutting down due to the post service exiting unexpectedly:
  • since the output of the post service is captured and added to the node logs it will show that connection/configuration problems were the cause

[poszu]

I have two ideas for improving the UX.

1. How about automatically enabling post GRPC service on the private services port when --start-smeshing=true and the post service is not already enabled?
2. Is there any reason to keep node-address field in the config? We already know in the code what value it should have. Perhaps we could just remove it and calculate it in the code to point to the address on which the post service is enabled on. Or leave it as an optional parameter with such a default - in case the user really needs to change its value to something else.

[fasmat]

How about automatically enabling post GRPC service on the private services port when --start-smeshing=true and the post service is not already enabled?

A PR with this change was already merged: #5259 the problem is a user might not want to use the private service but instead the TLS service (hybrid PoST setup: supervised + remote). We can say we do not want to support hybrid setup which would make things easier on our side but may interfere with plans by users...

Is there any reason to keep node-address field in the config? We already know in the code what value it should have. Perhaps we could just remove it and calculate it in the code to point to the address on which the post service is enabled on. Or leave it as an optional parameter with such a default - in case the user really needs to change its value to something else.

Theoretically node-address can be automatically set to the listener that exposes post adding https:// if it's TLS listener or http:// otherwise. We should however still add my suggestion to exit the post service if it fails to connect after a certain number of tries and have the node panic when the supervised service exits. Just to ensure that an invalid configuration is not ignored because it only complains in the logs.

[poszu]

Opened spacemeshos/post-rs#151 for the --max-retries.