[Core feature] Introduce a 'TERMINATING' workflow execution phase for aborting workflows

[katrogan]

Motivation: Why do you think this is important?

Currently, the process to abort an execution consists of two steps:

1. A user submits a terminate execution request which triggers the deletion of the Flyte workflow CRD. If provided, an abort reason is captured and saved in the db. However, only flytepropeller events can update a workflow execution phase on record so flyteadmin will continue to report the workflow in phase 'RUNNING'
2. After flytepropeller observes that the CRD has been deleted it will send an 'ABORTED' event at which point the workflow execution phase will officially update

In between these two steps the workflow appears to be in 'RUNNING' which is a confusing experience for users who would expect that their workflow should have been updated as a result of a successfully handled TerminateExecutionRequest (with a response of http 200/OK).

If instead, upon receiving the TerminateExecutionRequest flyteadmin updated the execution phase to something like 'TERMINATING' to indicate that the workflow is in the process of being aborted, this better reflects the underlying state of affairs.

Goal: What should the final outcome look like, ideally?

The immediate outcome of a call to terminate a workflow execution should reflect some status update to suggest that the request is being processed.

Describe alternatives you've considered

Step 1. above can't immediately update the workflow execution phase to 'ABORTED'. Existing in-progress tasks could still be running until flytepropeller properly picks up on the deleted flyte workflow CRD

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[katrogan]

cc @kumare3 @jsonporter