Flows get stuck in Running state

[tsteijvers]

First check

• I added a descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the Prefect documentation for this issue.
• I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

If during a flow run the k8s pod gets evicted, the flow run will stay in Running mode forever. We have had similar issues a few months ago but they seemed resolved (the flow run would get marked as Crashed). Now, since about 2-3 weeks we're seeing stuck flow runs again and we have to manually cancel then to get them out of the running state.

This is a major blocker for us since our flows need to run on a tight scheduled basis and no 2 flows can run simultaneously. So stuck flow runs will block our entire process.

Reproduction

# Start a flow run
# Ensure the pod gets evicted
# Flow will stay in Running state

Error

No response

Versions

Version:             2.10.18
API version:         0.8.4
Python version:      3.9.17
Git commit:          cf177852
Built:               Thu, Jun 29, 2023 2:34 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.40.1

Additional context

No response

[zangell44]

This should be resolved in release 2.11.4. Duplicate of #9246