Fix retries

[Tom-Newton]

Purpose of this PR

Fix retries for submission failure and application. Closes: #2237 and apparently maybe #2118 too.

Proposed changes:

• Convert isNextRetryDue into timeUntilNextRetryDue so we know how to set RequeueAfter.
• After completing a reconcile where the outcome is to wait longer before retrying, make sure to set a requeue time. Without this the controller won't attempt to reconcile the SparkApplication again until 10 hours later after default Cache.SyncPeriod.
• Remove the bool return from runSparkSubmit and just return error, because the bool argument is equivalent to err != nil. Also remove the related code in submitSparkApplication since that now seems to be out of date.
• Refactor submitSparkApplication and how its called a bit to ensure that submission and execution attempt counts, logging of errors and state are updated consistently.
• Always return an error with meaningful name in reconcileFailedSubmissionSparkApplication if validateSparkResourceDeletion returns false. Previously if was returning err set from a prior and possibly unrelated function call.
• Added new e2e tests for submission failure and application failure.

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly. - I don't think any change is required
• I have added tests that prove my changes are effective or that my feature works
• Existing unit tests pass locally with my changes.

Additional Notes

I'm a golang noob

[Tom-Newton]

I wasn't expecting to get reviewers assigned after opening as draft. This isn't ready for review yet.

[jacobsalway]

Thanks for the tag. Sorry have been busy with work but will review this tonight.

[josecsotomorales]

@ChenYi015 can we review this PR and create a new release if it looks good? @jacobsalway I'm happy to help with contributions

[josecsotomorales]

Hey @ChenYi015, it looks like change requests were addressed, can we get another review?

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ChenYi015]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ChenYi015]

/lgtm

[ChenYi015]

@josecsotomorales Have released v2.1.0-rc.0 with this PR included.