Request: detect when jobs are blocked from starting due to k8s limits

[ShardPhoenix]

Description

My k8s namespace has resource limits (max total CPU and RAM). If starting a Prefect job pod would go over those limits, it just never starts, and the flow on the UI hangs indefinitely. The agent logs still say Completed deployment of flow run. When I increased the limits, the job pod started immediately and the flow completed as normal.

Expected Behavior

It would be nice if this scenario could be detected and result in a useful error on the UI or agent level.

[chrisguidry]

I was able to reproduce this on a local minikube running Prefect's Kubernetes Worker by adding a ResourceQuota to the namespace:

namespace-quotas.yaml 
apiVersion: v1
kind: ResourceQuota
metadata:
  namespace: prefect
  name: mem-cpu-quota
spec:
  hard:
    requests.cpu: "250m"
    requests.memory: 50Mi
    limits.cpu: "500m"
    limits.memory: 100Mi

My work pool doesn't specify cpu/memory requests for the pods it creates, so the flow run is stuck in Pending, and what's concerning is that there's no indication of an error from my worker's logs.

[chrisguidry]

Correction, after 5 minutes (my pod watch timeout), I did get a notice that the flow run had crashed because the pod never started:

prefect-worker-774d5b666-7smb2 prefect-worker 18:16:45.799 | ERROR   | prefect.flow_runs.worker - Job 'classic-limpet-cmmkp': Pod never started.
prefect-worker-774d5b666-7smb2 prefect-worker 18:16:45.937 | INFO    | prefect.flow_runs.worker - Reported flow run '07e54c68-2098-4c0b-accd-f71b44d23e7d' as crashed: Flow run infrastructure exited with non-zero status code -1.

[chrisguidry]

With the Kubernetes worker, we are getting the correct behavior we want. If a pod can't be scheduled, after the pod watch timeout for the work pool, the flow will be reported as crashed. The relevant logs from the worker will be included as well.

However, the logs are exceedingly vague:

Flow run infrastructure exited with non-zero status code -1.

For pods that couldn't be scheduled at all

or

Flow run infrastructure exited with non-zero status code 137.

For pods that start but then OOM

I'd like to expose more details from the Kubernetes worker when there's a scheduling failure, if possible

[chrisguidry]

[Note, this issue should move to the prefect-kubernetes collection]