Backport of Stop processing ACME verifications when active node is stepped down into release/1.14.x

[hc-github-team-secure-vault-core]

Backport

This PR is auto-generated from #23278 to be assessed for backporting due to the inclusion of the label backport/1.14.x.

The below text is copied from the body of the original PR.

---

• Do not load existing ACME challenges persisted within storage on non-active nodes. This was the main culprit of the issues, secondary nodes would load existing persisted challenges trying to resolve them but writes would fail leading to the excessive logging.

  • We now handle this by not starting the ACME background thread on non-active nodes, while also checking within the scheduling loop and breaking out. That will force a re-reading of the Closing channel that should have been called by the PKI plugin's Cleanup method.

• If a node is stepped down from being the active node while it is actively processing a verification, we could get into an infinite loop due to an ErrReadOnly error attempting to clean up a challenge entry

• Add a maximum number of retries for errors around attempting to decode,fetch challenge/authorization entries from disk. We use double the number of "normal" max attempts for these types of errors, than we would for normal ACME retry attempts to avoid collision issues. Note that these additional retry attempts are not persisted to disk and will restart on every node start

• Add a 1 second backoff to any disk related error to not immediately spin on disk/io errors for challenges.

---

Overview of commits

• dbfaa6f

[github-actions]

Build Results:
All builds succeeded! ✅

[github-actions]

CI Results:
All Go tests succeeded! ✅