Create KubernetesEventsReplicator and replicate events while watching jobs

[bunchesofdonald]

This creates a class called KubernetesEventsReplicator, this class spins up a thread and watches the event stream from kubernetes and replicates pod events to Prefect Events.

Example

Here are two example events, the first is a pod 'running' event, which is the same structure that everything except the 'evicted' event:

{
  "event": "prefect.kubernetes.pod.running",
  "related": [
    {
      "prefect.resource.id": "prefect.deployment.8eb5492d-ebcc-44ac-878a-40d07d98aae7",
      "prefect.resource.name": "happy_little_tree",
      "prefect.resource.role": "deployment"
    },
    {
      "prefect.resource.id": "prefect.flow.b0478707-80b6-45ec-88fa-5fa4bfd45980",
      "prefect.resource.name": "happy",
      "prefect.resource.role": "flow"
    },
    {
      "prefect.resource.id": "prefect.flow-run.f02a1a9a-c989-42e6-9cd4-dad616ed6ec9",
      "prefect.resource.name": "granite-dodo",
      "prefect.resource.role": "flow-run"
    },
    {
      "prefect.resource.id": "prefect.work-pool.b96987f9-5141-4c16-b40c-564190baddcf",
      "prefect.resource.name": "my-k8s-pool",
      "prefect.resource.role": "work-pool"
    },
    {
      "prefect.version": "2.10.5",
      "prefect.resource.id": "prefect.worker.kubernetes.k8s-worker",
      "prefect.worker-type": "kubernetes",
      "prefect.resource.name": "k8s-worker",
      "prefect.resource.role": "worker"
    }
  ],
  "resource": {
    "prefect.resource.id": "prefect.kubernetes.pod.010e3a06-a13c-47a4-bb6f-82974e7136b6",
    "kubernetes.namespace": "default",
    "prefect.resource.name": "granite-dodo-rcjpt-lnbmw"
  },
  "workspace": "d3a6209b-65a4-465e-b018-8b2433f657be"
}

The second is the evicted event, which is very similar, except that it adds a kubernetes.reason to the resource:

{
  ...
  "event": "prefect.kubernetes.pod.evicted",
  "related": [
    {
      "prefect.resource.id": "prefect.deployment.8eb5492d-ebcc-44ac-878a-40d07d98aae7",
      "prefect.resource.name": "happy_little_tree",
      "prefect.resource.role": "deployment"
    },
    {
      "prefect.resource.id": "prefect.flow.b0478707-80b6-45ec-88fa-5fa4bfd45980",
      "prefect.resource.name": "happy",
      "prefect.resource.role": "flow"
    },
    {
      "prefect.resource.id": "prefect.flow-run.200a8804-60b7-403a-b89d-c6995f88cce2",
      "kubernetes.namespace": "default",
      "prefect.resource.name": "venomous-tody",
      "prefect.resource.role": "flow-run"
    },
    {
      "prefect.resource.id": "prefect.work-pool.b96987f9-5141-4c16-b40c-564190baddcf",
      "prefect.resource.name": "my-k8s-pool",
      "prefect.resource.role": "work-pool"
    },
    {
      "prefect.version": "2.10.5",
      "prefect.resource.id": "prefect.worker.kubernetes.k8s-worker",
      "prefect.worker-type": "kubernetes",
      "prefect.resource.name": "k8s-worker",
      "prefect.resource.role": "worker"
    }
  ],
  "resource": {
    "kubernetes.reason": "OOMKilled",
    "prefect.resource.id": "prefect.kubernetes.pod.ce5a56bd-8454-425b-a05a-f222b3ae6a02",
    "kubernetes.namespace": "default",
    "prefect.resource.name": "venomous-tody-95m62-52v96"
  }
}

[chrisguidry]

🤩

[chrisguidry]

👨‍🍳

[pleek91]

@bunchesofdonald resource labels look good to me!

[desertaxle]

🚀