Failed to change internal tls from true to false #640

heww · 2021-05-11T10:55:29Z

2021-05-11T10:52:23.783Z	ERROR	controller	Reconciler error	{"reconcilerGroup": "goharbor.io", "reconcilerKind": "Core", "controller": "core", "name": "harborcluster-sample-harbor-harbor", "namespace": "cluster-sample-ns", "error": "cannot set status to error: cannot set conditions to error: apply apps/v1, Kind=Deployment (cluster-sample-ns/harborcluster-sample-harbor-harbor-core): apply: Deployment.apps \"harborcluster-sample-harbor-harbor-core\" is invalid: [spec.template.spec.volumes[5].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[5].name: Not found: \"internal-certificates\"]: Deployment.apps \"harborcluster-sample-harbor-harbor-core\" is invalid: [spec.template.spec.volumes[5].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[5].name: Not found: \"internal-certificates\"]: apply apps/v1, Kind=Deployment (cluster-sample-ns/harborcluster-sample-harbor-harbor-core): apply: Deployment.apps \"harborcluster-sample-harbor-harbor-core\" is invalid: [spec.template.spec.volumes[5].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[5].name: Not found: \"internal-certificates\"]: Deployment.apps \"harborcluster-sample-harbor-harbor-core\" is invalid: [spec.template.spec.volumes[5].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[5].name: Not found: \"internal-certificates\"]", "errorVerbose": "Deployment.apps \"harborcluster-sample-harbor-harbor-core\" is invalid: [spec.template.spec.volumes[5].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[5].name: Not found: \"internal-certificates\"]: Deployment.apps \"harborcluster-sample-harbor-harbor-core\" is invalid: [spec.template.spec.volumes[5].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[5].name: Not found: \"internal-certificates\"]\napply\ngithub.com/goharbor/harbor-operator/pkg/controller.(*Controller).applyAndCheck\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/controller/common.go:167\ngithub.com/goharbor/harbor-operator/pkg/controller.(*Controller).ProcessFunc.func1\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/controller/resource.go:145\ngithub.com/goharbor/harbor-operator/pkg/graph.(*resourceManager).Run.func1\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/graph/runner.go:42\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/runtime/asm_amd64.s:1374\napply apps/v1, Kind=Deployment (cluster-sample-ns/harborcluster-sample-harbor-harbor-core)\ngithub.com/goharbor/harbor-operator/pkg/controller.(*Controller).ProcessFunc.func1\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/controller/resource.go:147\ngithub.com/goharbor/harbor-operator/pkg/graph.(*resourceManager).Run.func1\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/graph/runner.go:42\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/runtime/asm_amd64.s:1374\ncannot set status to error: cannot set conditions to error: apply apps/v1, Kind=Deployment (cluster-sample-ns/harborcluster-sample-harbor-harbor-core): apply: Deployment.apps \"harborcluster-sample-harbor-harbor-core\" is invalid: [spec.template.spec.volumes[5].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[5].name: Not found: \"internal-certificates\"]: Deployment.apps \"harborcluster-sample-harbor-harbor-core\" is invalid: [spec.template.spec.volumes[5].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[5].name: Not found: \"internal-certificates\"]\ngithub.com/goharbor/harbor-operator/pkg/controller.(*Controller).HandleError\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/controller/errors.go:64\ngithub.com/goharbor/harbor-operator/pkg/controller.(*Controller).Reconcile\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/controller/common.go:150\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/runtime/asm_amd64.s:1374"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
W0511 10:53:03.947141       1 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

The text was updated successfully, but these errors were encountered:

bitsf · 2021-05-18T13:23:53Z

it seems we are using c.Client.Patch() in pkg/controller/apply.go:31 , to apply the deployment, which will merge two different type volumes in this case.
@holyhope can we change this to c.Client.Update()

steven-zou · 2021-05-27T07:40:07Z

2021-05-27T07:37:32.330Z ERROR harbor-operator.controller Cannot deploy resource {"controller": "core", "version": "v1.0.0", "git.commit": "53ffaeae2c5406a5a9f7da9c9a787b28dc21362b", "request": "cluster-sample-ns/harborcluster-sample-harbor-harbor", "resource.apiVersion": "apps/v1", "resource.kind": "Deployment", "resource.name": "harborcluster-sample-harbor-harbor-core", "resource.namespace": "cluster-sample-ns", "error": "Deployment.apps "harborcluster-sample-harbor-harbor-core" is invalid: [spec.template.spec.volumes[6].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[6].name: Not found: "internal-certificates"]"}
github.com/goharbor/harbor-operator/pkg/controller.(*Controller).applyAndCheck
/home/runner/work/harbor-operator/harbor-operator/pkg/controller/common.go:165
github.com/goharbor/harbor-operator/pkg/controller.(*Controller).ProcessFunc.func1
/home/runner/work/harbor-operator/harbor-operator/pkg/controller/resource.go:145
github.com/goharbor/harbor-operator/pkg/graph.(*resourceManager).Run.func1
/home/runner/work/harbor-operator/harbor-operator/pkg/graph/runner.go:42
golang.org/x/sync/errgroup.(*Group).Go.func1
/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
2021-05-27T07:37:32.331Z ERROR controller Reconciler error {"reconcilerGroup": "goharbor.io", "reconcilerKind": "Core", "controller": "core", "name": "harborcluster-sample-harbor-harbor", "namespace": "cluster-sample-ns", "error": "cannot set status to error: cannot set conditions to error: apply apps/v1, Kind=Deployment (cluster-sample-ns/harborcluster-sample-harbor-harbor-core): apply: Deployment.apps "harborcluster-sample-harbor-harbor-core" is invalid: [spec.template.spec.volumes[6].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[6].name: Not found: "internal-certificates"]: Deployment.apps "harborcluster-sample-harbor-harbor-core" is invalid: [spec.template.spec.volumes[6].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[6].name: Not found: "internal-certificates"]: apply apps/v1, Kind=Deployment (cluster-sample-ns/harborcluster-sample-harbor-harbor-core): apply: Deployment.apps "harborcluster-sample-harbor-harbor-core" is invalid: [spec.template.spec.volumes[6].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[6].name: Not found: "internal-certificates"]: Deployment.apps "harborcluster-sample-harbor-harbor-core" is invalid: [spec.template.spec.volumes[6].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[6].name: Not found: "internal-certificates"]", "errorVerbose": "Deployment.apps "harborcluster-sample-harbor-harbor-core" is invalid: [spec.template.spec.volumes[6].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[6].name: Not found: "internal-certificates"]: Deployment.apps "harborcluster-sample-harbor-harbor-core" is invalid: [spec.template.spec.volumes[6].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[6].name: Not found: "internal-certificates"]\napply\ngithub.com/goharbor/harbor-operator/pkg/controller.(*Controller).applyAndCheck\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/controller/common.go:167\ngithub.com/goharbor/harbor-operator/pkg/controller.(*Controller).ProcessFunc.func1\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/controller/resource.go:145\ngithub.com/goharbor/harbor-operator/pkg/graph.(*resourceManager).Run.func1\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/graph/runner.go:42\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/runtime/asm_amd64.s:1374\napply apps/v1, Kind=Deployment (cluster-sample-ns/harborcluster-sample-harbor-harbor-core)\ngithub.com/goharbor/harbor-operator/pkg/controller.(*Controller).ProcessFunc.func1\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/controller/resource.go:147\ngithub.com/goharbor/harbor-operator/pkg/graph.(*resourceManager).Run.func1\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/graph/runner.go:42\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/runtime/asm_amd64.s:1374\ncannot set status to error: cannot set conditions to error: apply apps/v1, Kind=Deployment (cluster-sample-ns/harborcluster-sample-harbor-harbor-core): apply: Deployment.apps "harborcluster-sample-harbor-harbor-core" is invalid: [spec.template.spec.volumes[6].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[6].name: Not found: "internal-certificates"]: Deployment.apps "harborcluster-sample-harbor-harbor-core" is invalid: [spec.template.spec.volumes[6].secret: Forbidden: may not specify more than 1 volume type, spec.template.spec.containers[0].volumeMounts[6].name: Not found: "internal-certificates"]\ngithub.com/goharbor/harbor-operator/pkg/controller.(*Controller).HandleError\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/controller/errors.go:64\ngithub.com/goharbor/harbor-operator/pkg/controller.(*Controller).Reconcile\n\t/home/runner/work/harbor-operator/harbor-operator/pkg/controller/common.go:150\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.15.11/x64/src/runtime/asm_amd64.s:1374"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
/home/runner/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90

steven-zou · 2021-05-27T08:41:10Z

The patch strategy for deployment.podspec.volumes is patch strategy: merge,retainKeys, check details： https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#podspec-v1-core

steven-zou · 2021-05-27T09:02:30Z

Patch strategy is merge, so it does not support replacing container which contains Envs and volumeMounts .

Patch strategy for Envs and volumeMounts are all merge.

holyhope · 2021-05-27T10:52:03Z

For this issue only:
https://github.com/goharbor/harbor-operator/blob/master/controllers/goharbor/core/deployments.go#L398-L407
In the else case:

} else {
    envs = append(envs, corev1.EnvVar{
        Name:  "INTERNAL_TLS_TRUST_CA_PATH",
        Value: "",
    }, corev1.EnvVar{
        Name:  "INTERNAL_TLS_CERT_PATH",
        Value: "",
    }, corev1.EnvVar{
        Name:  "INTERNAL_TLS_KEY_PATH",
        Value: "",
    })
}

The merge strategy on the envs field will delete (set a empty value) to those environment variables.

But the issue of merging fields is more global and should (also) be handled at an upper level.

Delete fields from the spec

This one is specific to merge strategy of each fields.
ATM. I do not know how to handle it in a global way.
Maybe thanks to the DiscoveryAPI? (Seems like using a massive destruction weapon).
Delete the whole resource

I have suggestions about object that one:
1. Add a deleted object to the graph, instead of nil object:
  
  harbor-operator/controllers/goharbor/harbor/internaltls.go
  
  Line 80 in 852a3d0
  
  return nil, nil
2. List all resources (owned by the controller) and remove the one not in the graph.

bitsf · 2021-08-17T07:16:12Z

this issue still have problem, that component core still have spec.tls when changing internal tls from true to false, it will make core listen on 8443 port which should be 80 port

steven-zou · 2021-08-24T12:28:56Z

duplicated with #743

steven-zou added kind/bug Something isn't working release/1.0.1 area/reconciler reconciler/harbor labels May 11, 2021

steven-zou assigned bitsf May 13, 2021

bitsf mentioned this issue May 19, 2021

fix resources update #661

Closed

steven-zou added release/1.2 and removed release/1.0.1 labels May 27, 2021

bitsf mentioned this issue Jun 4, 2021

upgrade certmanger k8s version in CI #665

Merged

steven-zou closed this as completed in #665 Jun 15, 2021

steven-zou added release/1.1 and removed release/1.2 labels Jul 21, 2021

bitsf reopened this Aug 17, 2021

bitsf mentioned this issue Aug 17, 2021

apply failed after changing internal tls from false to true #743

Closed

steven-zou added known-issue Issues we have known but no solution or plan to introduce fix yet and removed release/1.1 labels Aug 20, 2021

steven-zou closed this as completed Aug 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to change internal tls from true to false #640

Failed to change internal tls from true to false #640

heww commented May 11, 2021

bitsf commented May 18, 2021

steven-zou commented May 27, 2021

steven-zou commented May 27, 2021 •

edited

Loading

steven-zou commented May 27, 2021

holyhope commented May 27, 2021

Delete fields from the spec

Delete the whole resource

bitsf commented Aug 17, 2021

steven-zou commented Aug 24, 2021

Failed to change internal tls from true to false #640

Failed to change internal tls from true to false #640

Comments

heww commented May 11, 2021

bitsf commented May 18, 2021

steven-zou commented May 27, 2021

steven-zou commented May 27, 2021 • edited Loading

steven-zou commented May 27, 2021

holyhope commented May 27, 2021

Delete fields from the spec

Delete the whole resource

bitsf commented Aug 17, 2021

steven-zou commented Aug 24, 2021

steven-zou commented May 27, 2021 •

edited

Loading