Add a delay after halt and before backup.

[dwedul-figure]

Summary

In Cosmovisor, allow a node operator to define a delay between the node halt (for upgrade) and backup.

Problem Definition

As a node operator, I want to define a delay after the halt (for upgrade) so that all node sub-processes have time to terminate and close file handles before moving on to steps that might need to access the files (e.g. backup or restart).

Proposal

Use a DAEMON_RESTART_DELAY environment variable to provide the desired delay as a time.Duration format (e.g. 5s or 300ms), default to 0.
As soon as the node halts, have Cosmovisor sleep for that amount of time before moving on to any subsequent steps.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[0xruff]

I second that. During some upgrades via cosmovisor I had problems with it probably restarting too fast. Cosmovisor or rather the binary it runs raises an error at these occasions that the database (leveldb/cleveldb) is still locked while trying to start a new process. After a manual restart of cosmovisor the nodes run just fine.
AFAIK cosmovisor itself does not know anything about locked files or database access of the process it runs, so the proposed delay is a good minimalistic approach to tackle the problem imo.

Logs of described issue:

May 27 xx:xx:xx <host> cosmovisor[86693]: 12:02AM INF daemon shutting down in an attempt to restart module=cosmovisor
May 27 xx:xx:xx <host> cosmovisor[86693]: 12:02AM INF pre-upgrade command does not exist. continuing the upgrade. module=cosmovisor
May 27 xx:xx:xx <host> cosmovisor[86693]: 12:02AM INF upgrade detected, relaunching app=provenanced module=cosmovisor
May 27 xx:xx:xx <host> cosmovisor[86693]: 12:02AM INF running app args=["start"] module=cosmovisor path=/home/<user>/.provenanced/cosmovisor/upgrades/lava/bin/provenanced
May 27 xx:xx:xx <host> cosmovisor[2110771]: Error: could not create "cleveldb" db in /home/<user>/.provenanced/data with name "application": failed to initialize database: IO error: lock /home/<user>/.provenanced/data/application.db/LOCK: Resource temporarily unavailable
May 27 xx:xx:xx <host> cosmovisor[86693]: 12:02AM ERR  error="exit status 1" module=cosmovisor```

[ValarDragon]

Oh wow, this is a great point that I haven't thought about before

I support this being added