Support registryUrls using regex manager

[ChipWolf]

Which Renovate are you using?

WhiteSource Renovate App

Which platform are you using?

GitHub.com

What would you like to do?

I'm attempting to use the regexManager with the Helm datasource to capture the required values from a Kubernetes HelmRelease CRD in order to select the correct registry, chart name, and version.
So far, I've successfully captured the depName and currentValue, but the registryUrl needs to be specified as a list as per this section of the docs.

I'm not sure how to capture the repository URL as a list using matchStrings, and would appreciate any help in achieving this.

Example HelmRelease CRD:

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: prometheus-operator
  namespace: monitoring
spec:
  releaseName: prometheus-operator
  chart:
    repository: https://kubernetes-charts.storage.googleapis.com/
    name: prometheus-operator
    version: 8.12.13
...

Current progress with renovate.json:

{
  "extends": ["config:base"],
  "regexManagers": [
    {
      "fileMatch": ["\\.yaml$"],
      "matchStrings": [
        "chart:\n *repository: (?<registryUrls>.*?)\n *name: (?<depName>.*?)\n *version: (?<currentValue>.*)\n"
      ],
      "datasourceTemplate": "helm"
    }
  ]
}

[rarkins]

This is not currently possible, so I moved it as a feature request into the main repo. Because registryUrls is technically an array, we need to think how to structure this. One possibility would be to initially support only a single registryUrl and then convert that into an array of length 1 afterwards.

[ChipWolf]

@rarkins thanks for the super quick response.
My initial thought would be to do this for each known repository in the meantime:

{
  "extends": ["config:base"],
  "regexManagers": [
    {
      "fileMatch": ["\\.yaml$"],
      "matchStrings": [
        "chart:\n *repository: https://kubernetes-charts.storage.googleapis.com/\n *name: (?<depName>.*?)\n *version: (?<currentValue>.*)\n"
      ],
      "datasourceTemplate": "helm",
      "registryUrls": ["https://kubernetes-charts.storage.googleapis.com/"]
    }
  ]
}

I imagine a more permanent solution could be to add a template for registries?

{
  "extends": ["config:base"],
  "regexManagers": [
    {
      "fileMatch": ["\\.yaml$"],
      "matchStrings": [
        "chart:\n *repository: (?<registry>.*?)\n *name: (?<depName>.*?)\n *version: (?<currentValue>.*)\n"
      ],
      "datasourceTemplate": "helm",
      "registryUrlsTemplate": ["{{registry}}"]
    }
  ]
}

[rarkins]

I think it might work if you take the registryUrls part out and put it into a packageRule instead.

[ChipWolf]

Could you possibly provide an example for that please? I'm new to this tool. Thanks! @rarkins

[rarkins]

{
  "extends": ["config:base"],
  "regexManagers": [
    {
      "fileMatch": ["\\.yaml$"],
      "matchStrings": [
        "chart:\n *repository: https://kubernetes-charts.storage.googleapis.com/\n *name: (?<depName>.*?)\n *version: (?<currentValue>.*)\n"
      ],
      "datasourceTemplate": "helm",
    }
  ],
  "packageRules": [{
    "datasources": ["helm"],
    "managers": ["regex"],
    "registryUrls": ["https://kubernetes-charts.storage.googleapis.com/"]
  }]
}

[ChipWolf]

Thanks for that. So, what if there's multiple registries for different HelmReleases? How would I pin that packageRule to a specific regexManager?

[rarkins]

You would need to write multiple package rules and match based on packageNames or something like that.

[ChipWolf]

This config is working for me if anyone reading this is interested. The first regexManager is capturing HelmReleases which use chart repositories (specified in packageRules), the second is capturing any HelmReleases using GitHub repos and updating based on tags.

{
  "extends": ["config:base"],
  "kubernetes": {
    "fileMatch": ["\\.yaml$"]
  },
  "regexManagers": [
    {
      "fileMatch": ["\\.yaml$"],
      "matchStrings": [
        "chart:\n *repository: .*?\n *name: (?<depName>.*?)\n *version: (?<currentValue>.*)\n"
      ],
      "datasourceTemplate": "helm"
    },
    {
      "fileMatch": ["\\.yaml$"],
      "matchStrings": [
        "chart:\n *git: [email protected]:(?<depName>.*?).git\n *ref: (?<currentValue>.*?)\n"
      ],
      "datasourceTemplate": "github-tags"
    }
  ],
  "packageRules": [
    {
      "datasources": ["helm"],
      "managers": ["regex"],
      "packageNames": [
        "sealed-secrets",
        "kube-ops-view",
        "metrics-server",
        "prometheus-operator"
      ],
      "registryUrls": ["https://kubernetes-charts.storage.googleapis.com/"]
    },
    {
      "datasources": ["helm"],
      "managers": ["regex"],
      "packageNames": [
        "ingress-nginx"
      ],
      "registryUrls": ["https://kubernetes.github.io/ingress-nginx"]
    },
    {
      "datasources": ["helm"],
      "managers": ["regex"],
      "packageNames": [
        "external-dns"
      ],
      "registryUrls": ["https://charts.bitnami.com/bitnami"]
    }
  ]
}

[RichiCoder1]

Would love to see this, especially as more and more charts are served from the company's repository and not the stable or bitnami repos.

Edit: for context, with ChipWolf's example I have 8 package rules, not counting stable, with at least two more on the way.

[Djiit]

I'd like to see this supported too. Would it be possible to extract the actual repo URL from the yaml file?
I'm willing to help with a little guidance :)

[rarkins]

Would support for a single registryUrl satisfy your current use cases?

[rarkins]

Also, are you only using the regex manager because if a deficiency in our current Helm or k8s package manager support?

[RichiCoder1]

@rarkins Single registryUrl would work for us, and we're suing regex because we're using Helm Operator (and soon Helm Controller) to handle Helm releases. Adding native support for those to Dependabot's Helm support would also work for us, but might not address other abstractions.

[viceice]

@RichiCoder1 I've added a new isue to track the Helm Operator manager. Should be easy to implement, as it's only some simple yaml.

[Djiit]

Would support for a single registryUrl satisfy your current use cases?

A single registryUrl per chart, yes, not a single one for all the charts.

[ChipWolf]

@Djiit does this not work for you? #6130 (comment)

[Djiit]

@ChipWolf Yes! I'd just like to be able to get rid of the redundancy (redeclare registries in renovate.json)

[RichiCoder1]

^ Yup. It works, but it's not ideal as I mentioned here: #6130 (comment)

[onedr0p]

Also showing my interest in this there's no doubt @ChipWolf method works but having to redeclare all helm chart sources and apps can get pretty hairy.

It's also worth noting that Flux probably nearing EOL and being replaced with Flux2 in the future.

Defining a HelmRepository in Flux2:

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: HelmRepository
metadata:
  name: kubernetes-sigs-descheduler-charts
  namespace: flux-system
spec:
  interval: 10m
  url: https://kubernetes-sigs.github.io/descheduler
  timeout: 3m

Sample of a Flux2 HelmRelease:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: descheduler
  namespace: kube-system
spec:
  interval: 5m
  chart:
    spec:
      chart: descheduler-helm-chart
      version: 0.19.0
      sourceRef:
        kind: HelmRepository
        name: kubernetes-sigs-descheduler-charts
        namespace: flux-system
      interval: 5m
  values:
    image:
      repository: k8s.gcr.io/descheduler/descheduler
      tag: v0.19.0
      pullPolicy: IfNotPresent
    deschedulerPolicy:
      strategies:
        RemoveDuplicates:
          enabled: false
        RemovePodsViolatingNodeAffinity:
          enabled: true
          params:
            nodeAffinityType:
            - requiredDuringSchedulingIgnoredDuringExecution
        RemovePodsViolatingInterPodAntiAffinity:
          enabled: false
        LowNodeUtilization:
          enabled: false

[ChipWolf]

The hack with fluxv2 which worked for me was to put the registry URL commented out above the chart name; not ideal, but it works

[onedr0p]

heh good idea @ChipWolf :D

[renovate-release]

🎉 This issue has been resolved in version 23.85.0 🎉

The release is available on:

• GitHub release
• 23.85.0

Your semantic-release bot 📦🚀

[Djiit]

Hey @rarkins, thank for the PR! Any quick example of how we are supposed to use it ? Thanks

[rarkins]

@Djiit this line was added to the docs:

Does that clarify?

[Djiit]

Oh allright. This would work, right ?

  "regexManagers": [
    {
      "fileMatch": ["\\.yaml$"],
      "matchStrings": [
        "chart:\n *repository: (?<registryUrl>.*?)\n *name: (?<depName>.*?)\n *version: (?<currentValue>.*)\n"
      ],
      "datasourceTemplate": "helm"
    }
  ],

[rarkins]

Yes, we decided to keep it simple and not have a registryUrlTemplate field unless it's unavoidable for someone