Add time to live for finished jobs in Kubernetes #29439
Conversation
potiuk
force-pushed
the
fix-cleanup-of-jobs-for-terraform-redeployments
branch
from
1cecb63 to
294d266
Compare
potiuk
changed the title
When you upgrade/change job in K8S that has been finished and not manually removed, this leads to "Field is immutable" error. This is a known kubernetes issue: kubernetes/kubernetes#89657 And there are some workarounds (manually removing the job for example), but the only good solution is possible only in K8S 1.23+ with ttlSecondsAfterFinished set for the job, so that K8S can auto clean it. This PR adds it conditionally for K8S >= 1.23 Fixes: apache#21943
potiuk
force-pushed
the
fix-cleanup-of-jobs-for-terraform-redeployments
branch
from
294d266 to
645f29c
Compare
|
Hey @dstandish @jedcunningham -> WDYT? |
There was a problem hiding this comment.
It's an interesting change, but there's still a chance of getting the same error if for some reason we run the jobs twice before the TTL ends, on top of that we can add {{ .Release.Revision }} as suffix for job name -> metadata.name: {{ .Release.Name }}-<job name>-{{ .Release.Revision }}, and keep the cleanup task for garbage collection.
Thought about it, but I am not sure if there is another reason why we wanted to keep the job names "static" - @jedcunningham @dstandish ? |
|
Closing in favour of #29314 |
When you upgrade/change job in K8S that has been finished and not manually removed, this leads to "Field is immutable" error.
This is a known kubernetes issue:
kubernetes/kubernetes#89657
And there are some workarounds (manually removing the job for example), but the only good solution is possible only in K8S 1.23+ with ttlSecondsAfterFinished set for the job, so that K8S can auto
clean it.
This PR adds it conditionally for K8S >= 1.23
Fixes: #27561
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.