Loki CRDs missing on fresh install

[psschwei]

Steps to reproduce the problem

create new k8s cluster
cd infrastructure/helm/quantumserverless
helm install qs .

What is the current behavior?

Install fails with an error about missing CRDs:

$ helm install qs .
coalesce.go:223: warning: destination for loki.gateway.affinity is a table. Ignoring non-table value (podAntiAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels:
          {{- include "loki.gatewaySelectorLabels" . | nindent 10 }}
      topologyKey: kubernetes.io/hostname
)
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: [resource mapping not found for name: "loki" namespace: "" from "": no matches for kind "GrafanaAgent" in version "monitoring.grafana.com/v1alpha1"
ensure CRDs are installed first, resource mapping not found for name: "loki" namespace: "" from "": no matches for kind "LogsInstance" in version "monitoring.grafana.com/v1alpha1"
ensure CRDs are installed first, resource mapping not found for name: "loki" namespace: "" from "": no matches for kind "PodLogs" in version "monitoring.grafana.com/v1alpha1"
ensure CRDs are installed first]

What is the expected behavior?

Install succeeds.

I think this can be solved by tweaking some of the options in the helm chart...

[psschwei]

Yeah, installing the grafana-agent-operator by dropping https://github.com/Qiskit-Extensions/quantum-serverless/blob/cb13d40dddbbab974df1c932248c322dfaccc7c8/infrastructure/helm/quantumserverless/values.yaml#L360-L363 fixes it...

[psschwei]

Not ideal, but we could also install the CRDs manually before doing the helm install...

k create -f https://raw.githubusercontent.com/grafana/agent/main/production/operator/crds/monitoring.grafana.com_grafanaagents.yaml
k create -f https://raw.githubusercontent.com/grafana/agent/main/production/operator/crds/monitoring.grafana.com_logsinstances.yaml
k create -f https://raw.githubusercontent.com/grafana/agent/main/production/operator/crds/monitoring.grafana.com_podlogs.yaml

[psschwei]

@pacomf @IceKhan13 @akihikokuroda any preferences?

[akihikokuroda]

I've never seen this before. Do you know what has changed?

[psschwei]

not sure... the chart version was bumped in #392 , but there's no changelog so would have to go through 100s of commits to really check (if the version bump was even the cause...)

one thing I didn't mention in the original issue (though I alluded to it in the title) is that it happens on a new cluster... if you've been using the same cluster for a while, the CRDs may have already been installed before we dropped the grafana-agent-operator in #505

[pacomf]

no preferences... any solution that can fix the problem in the most easy way hahaha, @IceKhan13 thoughts?

[psschwei]

At some point in the future (i.e. not now), it might make sense to take a more holistic look at the observability stack... there's a lot of different ways to install the different components even just with the charts that we currently have (e.g. there's about 3-4 different ones that have a grafana option), so we may want to see if there's a more optimal way...

[psschwei]

Even easier solution: actually disable self-monitoring 😄

[akihikokuroda]

@psschwei I read it but I haven't tried it. Does it work?

[psschwei]

#572 fixes the problem by disabling the install of those components, so no need for the CRDs