Clusters-keeper: if a machine is pending for too long, it should terminate and retry a few times before giving up

[sanderegg]

In case an EC2 instance is "broken", i.e. not reachable it might never be able to run a docker stack.
In this case the clusters-keeper should give up on it after some X amount of time, terminate the instance and try again for Y times.
This use-case sometimes happen and the current behavior is that the machine is kept up forever.

Another variation is when the dask-scheduler on the primary machine somehow uses all the disk space available, then the docker swarm goes boom and the machine is lost.
In that case the clusters-keeper should also terminate the machines to spare money