vcjob should not set deleted state before all it's tasks still exists

[silenceli]

Hi, this maybe a bug.

problem

I list-and-watch vcjob url to monitor vcjob status changes, when I delete vcjob, I got the DELETED action message quickly. At the same time, the pods belong to this vcjob is still exist(in terminating status). The pods will still use the CPU/GPU/MEM resources, but volcano is already release the used resources.

I think this is a serious problem because it leads to a mess of used resources, and you must determine the status of the pods to know if the vcjob is really over.

how to solve

When vc-controller get delete cmd, It should delete all the pods belong to and set status to terminating. After all pods is deleted, then set vcjob deleted.

[hzxuzhonghu]

This aligns with k8s resource delete behavior like deployment. I think one way is to do “Foreground” deletion.

[silenceli]

This aligns with k8s resource delete behavior like deployment. I think one way is to do “Foreground” deletion.

Thank you! I will try “Foreground” deletion.

[silenceli]

params =  {
                "kind":"DeleteOptions",
                "apiVersion":"v1",
                "propagationPolicy":"Foreground"
            }

Add these params when delete can solve my problem, thanks！

[silenceli]

more info see:

https://kubernetes.io/zh/docs/concepts/workloads/controllers/garbage-collection/