Reapplying liquibase job in kubernetes failed

[paloliska]

Describe the bug

Using kubectl apply for deploying application with liquibase init job failed if already applied earlier with error:

The Job "quarkus-service-liquibase-mongodb-init" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"controller-uid":"***", "job-name":"quarkus-service-liquibase-mongodb-init"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume{core.Volume{Name:"google-cloud-key", VolumeSource:core.VolumeSource{HostPath:(*core.HostPathVolumeSource)(nil), EmptyDir:(*core.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*core.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*core.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*core.GitRepoVolumeSource)(nil), Secret:(*core.SecretVolumeSource)(0x********), NFS:(*core.NFSVolumeSource)(nil), ISCSI:(*core.ISCSIVolumeSource)(nil), Glusterfs:(*core.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*core.PersistentVolumeClaimVolumeSource)(nil), RBD:(*core.RBDVolumeSource)(nil), Quobyte:(*core.QuobyteVolumeSource)(nil), FlexVolume:(*core.FlexVolumeSource)(nil), Cinder:(*core.CinderVolumeSource)(nil), CephFS:(*core.CephFSVolumeSource)(nil), Flocker:(*core.FlockerVolumeSource)(nil), DownwardAPI:(*core.DownwardAPIVolumeSource)(nil), FC:(*core.FCVolumeSource)(nil), AzureFile:(*core.AzureFileVolumeSource)(nil), ConfigMap:(*core.ConfigMapVolumeSource)(nil), VsphereVolume:(*core.VsphereVirtualDiskVolumeSource)(nil), AzureDisk:(*core.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*core.PhotonPersistentDiskVolumeSource)(nil), Projected:(*core.ProjectedVolumeSource)(nil), PortworxVolume:(*core.PortworxVolumeSource)(nil), ScaleIO:(*core.ScaleIOVolumeSource)(nil), StorageOS:(*core.StorageOSVolumeSource)(nil), CSI:(*core.CSIVolumeSource)(nil), Ephemeral:(*core.EphemeralVolumeSource)(nil)}}}, InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"benefit-connector-liquibase-mongodb-init", Image:"eu.gcr.io/***:latest", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource{core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(0x********), SecretRef:(*core.SecretEnvSource)(nil)}, core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(nil), SecretRef:(*core.SecretEnvSource)(0x********)}, core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(0x********), SecretRef:(*core.SecretEnvSource)(nil)}, core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(0x********), SecretRef:(*core.SecretEnvSource)(nil)}}, Env:[]core.EnvVar{core.EnvVar{Name:"GOOGLE_APPLICATION_CREDENTIALS", Value:"/var/secrets/google/key.json", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"QUARKUS_LIQUIBASE_MONGODB_ENABLED", Value:"true", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:"QUARKUS_INIT_AND_EXIT", Value:"true", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, VolumeMounts:[]core.VolumeMount{core.VolumeMount{Name:"google-cloud-key", ReadOnly:false, MountPath:"/var/secrets/google", SubPath:"", MountPropagation:(*core.MountPropagationMode)(nil), SubPathExpr:""}}, VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc02072b640), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc00c93b4d0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable

Workaround is to run kubectl delete job/quarkus-service-liquibase-init before apply.

Expected behavior

• kubectl apply success
• There is io.quarkus.kubernetes.deployment.JobConfig.ttlSecondsAfterFinished property, which could be used to clean-up finished jobs as per spec: https://kubernetes.io/docs/concepts/workloads/controllers/job/#ttl-mechanism-for-finished-jobs

Actual behavior

deployment failed

How to Reproduce?

1. create and build project with liquibase or flyway extension, kubernetes extension
2. deploy application kubectl apply -f build/kubernetes/kubernetes.yml -> sucsess
3. deploy again -> failed

Output of uname -a or ver

Linux 6.1.0-1013-oem #13-Ubuntu SMP PREEMPT_DYNAMIC Thu May 18 16:45:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "17.0.7" 2023-04-18 OpenJDK Runtime Environment (build 17.0.7+7-Ubuntu-0ubuntu122.04.2) OpenJDK 64-Bit Server VM (build 17.0.7+7-Ubuntu-0ubuntu122.04.2, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.1.0.Final

Build tool (ie. output of mvnw --version or gradlew --version)

------------------------------------------------------------ Gradle 8.1.1 ------------------------------------------------------------ Build time: 2023-04-21 12:31:26 UTC Revision: 1cf537a851c635c364a4214885f8b9798051175b Kotlin: 1.8.10 Groovy: 3.0.15 Ant: Apache Ant(TM) version 1.10.11 compiled on July 10 2021 JVM: 17.0.7 (Private Build 17.0.7+7-Ubuntu-0ubuntu122.04.2) OS: Linux 6.1.0-1013-oem amd64

Additional information

No response

[quarkus-bot]

/cc @Sgitario (kubernetes), @andrejpetras (liquibase), @evanchooly (kotlin), @geoand (kotlin,kubernetes,liquibase), @gsmet (liquibase), @iocanel (kubernetes)

[geoand]

There is io.quarkus.kubernetes.deployment.JobConfig.ttlSecondsAfterFinished property, which could be used to clean-up finished jobs as per spec: https://kubernetes.io/docs/concepts/workloads/controllers/job/#ttl-mechanism-for-finished-jobs

I think we should probably do this by default

[paloliska]

Is there chance to push this forward? quarkus.kubernetes.externalize-init is true by default so user has to do action to mitigate this behaviour.

[paloliska]

Gently bringing up this issue. job is not possible to deploy 2nd time with quarkus 3.6.1

[gsmet]

@iocanel could you have a look at this one? It looks quite problematic. I know we discussed this behavior for some other features so maybe it has been solved.