Adds creation of sentinel files for checking provider status

[diverdane]

Desired Outcome

• SP creates an empty sentinel status file after initially providing secrets (either via secret files or Kubernetes Secrets). The SP creates this file at /conjur/status/CONJUR_SECRETS_PROVIDED, but this can be mounted by an application container via a shared volume mount at any arbitrary location in the container's file system.
• SP creates an empty sentinel status file whenever secret files or Kubernetes Secrets have been updated. The SP creates this file at /conjur/status/CONJUR_SECRETS_UPDATED, but this can be mounted by an application container via a shared volume mount at any arbitrary location in the container's file system.
• The SP includes a convenient script at /usr/local/bin/conjur_secrets_provided that waits for the .../CONJUR_SECRETS_PROVIDED to be created. This can be used in a postStart lifecycle hook definition for the SP container, to defer startup of app container until SP has completed its first round of providing secrets.
• At startup, the SP copies a script that monitors the .../CONJUR_SECRETS_UPDATED file to /conjur/status/conjur_secrets_unchanged. This script returns a non-zero exit status whenever secret files or Kubernetes Secrets have changed. This can be used in a livenessProbe or readinessProbe definition for an application container to force Kubernetes/kubelet to restart this container when secrets have been updated. The Deployment manifest would need to include a volumeMount to mount this directory/file in the application container, along with the livenessProbe / readinessProbe definition.

With the above changes, a postStart lifecycle hook for the Secrets Provider container would look like this:

        lifecycle:
          postStart:
            exec:
              command:
              - /usr/local/bin/conjur-secrets-provided

And a livenessProbe for an application container that would serve as a "file watcher" can potentially look something like this (assuming the livenessProbe is not already being used by the container as a health probe):

        livenessProbe:
          exec:
            command:
            - /mounted/status/conjur-secrets-unchanged
          failureThreshold: 1
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1

Where the application container (and SP container) would need to include volumeMounts similar to this:

        volumeMounts:
        - mountPath: /mounted/status
          name: conjur-status

and the Pod would need a Volume defined:

      volumes:
      - name: conjur-status
        emptyDir:
          medium: Memory

Implemented Changes

• SP creates an empty /conjur/status/CONJUR_SECRETS_PROVIDED file after initial round of providing secrets.
• SP creates an empty /conjur/status/CONJUR_SECRETS_UPDATED file whenever secret files or Kubernetes Secrets have been updated**.
• SP copies a conjur_secrets_unchanged script to /conjur/status in it file system. This directory can be volume mounted by the application container at any arbitrary path in the application container's file system.
• LOTS of refactoring to minimize the number of arguments passed to the public functions in pkt/secrets. Provider configuration was split up into separate embedded structs for:
  • Common provider config
  • Config specific to Push-to-File
  • Config specific to Kubernetes Secrets destination mode
• LOTS of refactoring to allow provider functions to return a boolean status indicating target files/Secrets have just been updated, along with an error, rather than just an error. This allows the top-level functions to create the .../CONJUR_SECRETS_UPDATED file when appropriate.
• LOTS of UT changes to account for refactoring and the new sentinel file functionality.

Connected Issue/Story

CyberArk internal issue link: ONYX-17885

Definition of Done

• Everything listed in implemented changes are implemented.
• UT is implemented for new code.

Changelog

• The CHANGELOG has been updated, or
• This PR does not include user-facing changes and doesn't require a
  CHANGELOG update

Test coverage

• This PR includes new unit and integration tests to go with the code
  changes, or
• The changes in this PR do not require tests

Documentation

• Docs (e.g. READMEs) were updated in this PR
• A follow-up issue to update official docs has been filed here: insert issue ID
• This PR does not require updating any documentation

Behavior

• This PR changes product behavior and has been reviewed by a PO, or
• These changes are part of a larger initiative that will be reviewed later, or
• No behavior was changed with this PR

Security

• Security architect has reviewed the changes in this PR,
• These changes are part of a larger initiative with a separate security review, or
• There are no security aspects to these changes

[szh]

Looking forward to the walk through. Here are a few initial minor comments.

[codeclimate]

Code Climate has analyzed commit 5ed806a and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 85.9% (50% is the threshold).

This pull request will bring the total coverage in the repository to 89.8% (-0.2% change).

View more on Code Climate.

[szh]

LGTM!