send webhook container halts if the callback secret is not present

[harshad16]

Describe the bug
The new callback feature component send-webhook container halts for callback secrets as it is volume mounted.
As the thamos sync based on user-api schema, the new thamos version would create default callback call on advise request.
However old version wouldn't provide callback call, which would cause this issue.

To Reproduce
Steps to reproduce the behavior:

1. Execute a thamos advise from older version of thamos.
2. Go to the cluster
3. Scroll down to send-webhook container.
4. See error

Expected behavior
Smooth transition of send-webhook.

Possible solutions

• Make the secret mount optional on adviser workflow. ex: here
  or
• Explore the option of having a condition on argo workflow to check if secrets exists, if not then don't mount.
  or
• create a dumpy secret entry for the callback-url and handle it process in workflow-helper and user-api.

Acceptance criteria

• Fix the adviser run from failing due to wait time on secret mount.
• Verify the adviser run execute properly.

[harshad16]

/sig devsecops
/priority important-soon

[harshad16]

/triage accepted