Add tenacity retries to run method

[gabcoyne]

In order to support a greater degree of resiliency in job creation we're adding tenacity to preserve existing signatures and provide retry functionality.

Closes #116

Example

@retry(
    stop=stop_after_attempt(MAX_ATTEMPTS),
    wait=wait_fixed(RETRY_MIN_DELAY_SECONDS)
    + wait_random(
        RETRY_MIN_DELAY_JITTER_SECONDS,
        RETRY_MAX_DELAY_JITTER_SECONDS,
    ),
    reraise=True,
)

Screenshots

Checklist

• [x ] References any related issue by including "Closes #" or "Closes ".
  • If no issue exists and your change is not a small fix, please create an issue first.
• Includes tests or only affects documentation.
• Passes pre-commit checks.
  • Run pre-commit install && pre-commit run --all locally for formatting and linting.
• Includes screenshots of documentation updates.
  • Run mkdocs serve view documentation locally.