Flux does not update image from purely numerical tags #2703

alastairs · 2019-12-19T18:43:22Z

Our Docker images are tagged with a simple numeric value by our build system. I've a HelmRelease defined with annotations as follows:

fluxcd.io/automated: true
filter.fluxcd.io/extractors: regexp:^\d+$

flux has successfully deployed this Helm release, Kubernetes has pulled the image and started the container succesfully. Flux has also identified that there are newer versions of the image available:

WORKLOAD                   CONTAINER   IMAGE                        CREATED
dev:deployment/extractors  extractors  our.azurecr.io/extractors
                                       |   154                      19 Dec 19 15:48 UTC
                                       |   153                      10 Dec 19 12:06 UTC
                                       |   152                      09 Dec 19 09:45 UTC
                                       |   151                      28 Nov 19 14:17 UTC
                                       '-> 150                      27 Nov 19 09:04 UTC
                                           149                      21 Nov 19 17:49 UTC
                                           b2bf9fa                  21 Nov 19 13:14 UTC

However, the deployed image is, as indicated in that output, still v150, rather than the newest v154. Based on a comment from @stefanprodan, I have also tried specifying a filter of regexp:^[0-9]*$ with the same result.

I am able to provide logs with a bit of guidance on what components to focus on,e tc.

Additional context
Add any other context about the problem here, e.g

Flux version: 1.16
Helm Operator version: 1.0.0-rc4
Kubernetes version: 1.15.4-gke.22
Git provider: GitHub (SaaS)
Container registry provider: Azure Container Registry

The text was updated successfully, but these errors were encountered:

2opremio · 2019-12-20T09:19:06Z

However, the deployed image is, as indicated in that output, still v150, rather than the newest v154

Can you show us the creation time and creation labels of both images?

2opremio · 2019-12-20T09:20:31Z

Also, can you show us the full HelRelease? (with the full annotations and values)

alastairs · 2019-12-20T10:02:53Z

@2opremio Thanks for your quick reply. I'm not sure what you mean by "creation label", sorry. Is the "created" column in the issue description different from what you mean by "creation time"?

The HelmRelease is as follows:

---
apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: extractors
  namespace: dev
  annotations:
    fluxcd.io/automated: "true"
    # Tell Flux which repository to watch
    repository.fluxcd.io/extractors: extractors
    # Tell Flux how to update the Docker tag
    filter.fluxcd.io/extractors: regexp:^[0-9]*$
spec:
  releaseName: vehement-bunny
  chart:
    git: [email protected]:OurOrg/our-repo
    path: charts/extractors
    ref: deploy/dev
  rollback:
    enable: true
  values:
    # The number of instances of the service to run.
    replicaCount: 1

    # Secret to reference for credentials, etc.
    secretName: vehement-bunny-extractors

    # The number of revisions of the deployment to keep. Each `helm upgrade` counts
    # as a deployment revision.
    revisionHistoryLimit: 2

    image:
      # The registry containing the our images. The image name is hard-
      # coded into this chart.
      registry: our.azurecr.io

      # The version of the image to pull from the registry. The image name is hard-
      # coded into this chart.
      tag: 150

      # Under what circumstances Kubernetes will pull the image for updates. If not
      # set here, will default to IfNotPresent.
      pullPolicy: IfNotPresent

      # The secret(s) containing the credentials for authenticating with the Docker
      # registry when pulling images.
      pullSecrets:
        - docker-registry

    rabbitmq:
      # The URI of a RabbitMQ message broker.
      uri: #redacted

    environment:
      #redacted
    persistence:
      #redacted

    # Override resource requests by uncommenting the below lines and adjusting them
    # as necessary, and remove the curly braces after 'resources:'.
    resources:
      limits:
       cpu: 500m
       memory: 750Mi
      requests:
       cpu: 100m
       memory: 500Mi

    # See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
    # This will be copied wholesale into the Deployment resource for this chart.
    nodeSelector: {}

    # See https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
    # This will be copied wholesale into the Deployment resource for this chart.
    tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "worker"
        effect: "NoSchedule"

    # See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
    # This will be copied wholesale into the Deployment resource for this chart.
    affinity:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 90
            preference:
              matchExpressions:
                - key: "role"
                  operator: "In"
                  values:
                    - "worker"

stefanprodan · 2019-12-20T10:08:59Z

Replace filter.fluxcd.io/extractors: regexp:^[0-9]*$ with filter.fluxcd.io/chart-image: regexp:^[0-9]*$ and please read the docs to see how automation works for HelmReleases http://docs.fluxcd.io/en/1.17.0/references/helm-operator-integration.html

hiddeco · 2019-12-20T10:12:33Z

In addition to @stefanprodan his comment, the repository.fluxcd.io/extractors annotation can also be removed.

alastairs · 2019-12-20T10:40:00Z

Replace filter.fluxcd.io/extractors: regexp:^[0-9]*$ with filter.fluxcd.io/chart-image: regexp:^[0-9]*$ and please read the docs to see how automation works for HelmReleases http://docs.fluxcd.io/en/1.17.0/references/helm-operator-integration.html

Thanks @stefanprodan. I'm finding this area extremely confusing; in fact, in #2663 you yourself seemed to recommend changing from a chart-image annotation to an alias annotation.

In addition to @stefanprodan his comment, the repository.fluxcd.io/extractors annotation can also be removed.

Thanks @hiddeco. I'm further confused, because in creating #2670, it was indicated that the repository.fluxcd.io/<alias> annotation was required in order for the equivalent registry/tag/filter annotations to take effect (now documented as a result of that PR). Annoyingly I can't now find the reference for that, either here on GitHub or in Slack.

stefanprodan · 2019-12-20T10:45:55Z

@alastairs the annotations should match the HelmRelease values structure, not the deployment one, the problem with your values is that the image is not specified so Flux can't update it.

values:
  registry: 
  image: 
  tag:

alastairs · 2019-12-20T12:07:45Z

So based on http://docs.fluxcd.io/en/1.17.0/references/helm-operator-integration.html#automated-image-detection, I think the structure should be:

values:
  image:
    registry: our.azurecr.io
    repository: extractors # not image
    tag: 150

Does that look right?

stefanprodan · 2019-12-20T12:12:24Z

Yes that looks ok to me

alastairs · 2019-12-20T12:52:26Z

Ok, still no luck. Here's the current state of play:

`HelmRelease`

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
  name: extractors
  namespace: dev
  annotations:
    fluxcd.io/automated: "true"
    # Tell Flux how to update the Docker tag
    filter.fluxcd.io/chart-image: regexp:^[0-9]*$
spec:
  releaseName: vehement-bunny
  chart:
    git: [email protected]:OurOrg/our-repo
    path: charts/extractors
    ref: deploy/dev
  rollback:
    enable: true
  values:
    # The number of instances of the service to run.
    replicaCount: 1

    # Secret to reference for credentials, etc.
    secretName: vehement-bunny-extractors

    # The number of revisions of the deployment to keep. Each `helm upgrade` counts
    # as a deployment revision.
    revisionHistoryLimit: 2

    image:
      # The registry containing our images. The image name is hard-coded into this chart.
      registry: our.azurecr.io

      # The version of the image to pull from the registry. The image name is hard-
      # coded into this chart.
      tag: 150

      # Used by flux only, to find newer tags for deployment. The image name is hard-
      # coded into this chart.
      repository: extractors

      # Under what circumstances Kubernetes will pull the image for updates. If not
      # set here, will default to IfNotPresent.
      pullPolicy: IfNotPresent

      # The secret(s) containing the credentials for authenticating with the Docker
      # registry when pulling images.
      pullSecrets:
        - docker-registry

    rabbitmq:
      # The URI of a RabbitMQ message broker.
      uri: # redacted

    environment:
      # redacted
    persistence:
      # redacted

    # Override resource requests by uncommenting the below lines and adjusting them
    # as necessary, and remove the curly braces after 'resources:'.
    resources:
      limits:
       cpu: 500m
       memory: 750Mi
      requests:
       cpu: 100m
       memory: 500Mi

    # See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
    # This will be copied wholesale into the Deployment resource for this chart.
    nodeSelector: {}

    # See https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
    # This will be copied wholesale into the Deployment resource for this chart.
    tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "worker"
        effect: "NoSchedule"

    # See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
    # This will be copied wholesale into the Deployment resource for this chart.
    affinity:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 90
            preference:
              matchExpressions:
                - key: "role"
                  operator: "In"
                  values:
                    - "worker"

`fluxctl list-workloads -ndev`

WORKLOAD                                   CONTAINER                IMAGE                               RELEASE   POLICY
dev:deployment/vehement-bunny-extractors   extractors               ch360-on.azurecr.io/extractors:150  ready        
dev:helmrelease/extractors                 chart-image              ch360-on.azurecr.io/extractors      DEPLOYED  automated

`fluxctl list-images -w dev:deployment/vehement-bunny-extractors`

WORKLOAD                                 CONTAINER   IMAGE                     CREATED
dev:deployment/vehement-bunny-extractors extractors  our.azurecr.io/extractors 
                                                     |   155                   20 Dec 19 10:39 UTC
                                                     |   154                   19 Dec 19 15:48 UTC
                                                     |   153                   10 Dec 19 12:06 UTC
                                                     |   152                   09 Dec 19 09:45 UTC
                                                     |   151                   28 Nov 19 14:17 UTC
                                                     '-> 150                   27 Nov 19 09:04 UTC
                                                         149                   21 Nov 19 17:49 UTC

`fluxctl list-images -w dev:helmrelease/extractors`

WORKLOAD                    CONTAINER    IMAGE           CREATED
dev:helmrelease/extractors  chart-image
                                         |   155         20 Dec 19 10:39 UTC
                                         |   154         19 Dec 19 15:48 UTC
                                         |   153         10 Dec 19 12:06 UTC
                                         |   152         09 Dec 19 09:45 UTC
                                         |   151         28 Nov 19 14:17 UTC
                                         |   150         27 Nov 19 09:04 UTC
                                         |   149         21 Nov 19 17:49 UTC
                                         '-> (untagged)  ?

2opremio · 2020-01-08T11:20:47Z

Does the chart in charts/extractors also define entries with the same structure as the following?

values:
    image:
      # The registry containing our images. The image name is hard-coded into this chart.
      registry: our.azurecr.io

      # The version of the image to pull from the registry. The image name is hard-
      # coded into this chart.
      tag: 150

      # Used by flux only, to find newer tags for deployment. The image name is hard-
      # coded into this chart.
      repository: extractors

stefanprodan · 2020-01-08T11:25:12Z

In YAML tag: 150 makes the value an integer and I think it breaks the parser, should be tag: "150"

alastairs · 2020-01-17T17:57:02Z

Thank you @stefanprodan and @2opremio for your replies. I suspect it is the YAML parsing issue, good spot. I'm not able to verify this fix, unfortunately, so please assume this is the correct resolution 😄

alastairs added blocked-needs-validation Issue is waiting to be validated before we can proceed bug labels Dec 19, 2019

hiddeco added question and removed blocked-needs-validation Issue is waiting to be validated before we can proceed bug labels Dec 20, 2019

2opremio closed this as completed Feb 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flux does not update image from purely numerical tags #2703

Flux does not update image from purely numerical tags #2703

alastairs commented Dec 19, 2019

2opremio commented Dec 20, 2019

2opremio commented Dec 20, 2019

alastairs commented Dec 20, 2019

stefanprodan commented Dec 20, 2019

hiddeco commented Dec 20, 2019

alastairs commented Dec 20, 2019

stefanprodan commented Dec 20, 2019

alastairs commented Dec 20, 2019

stefanprodan commented Dec 20, 2019

alastairs commented Dec 20, 2019

2opremio commented Jan 8, 2020

stefanprodan commented Jan 8, 2020

alastairs commented Jan 17, 2020

Flux does not update image from purely numerical tags #2703

Flux does not update image from purely numerical tags #2703

Comments

alastairs commented Dec 19, 2019

2opremio commented Dec 20, 2019

2opremio commented Dec 20, 2019

alastairs commented Dec 20, 2019

stefanprodan commented Dec 20, 2019

hiddeco commented Dec 20, 2019

alastairs commented Dec 20, 2019

stefanprodan commented Dec 20, 2019

alastairs commented Dec 20, 2019

stefanprodan commented Dec 20, 2019

alastairs commented Dec 20, 2019

HelmRelease

fluxctl list-workloads -ndev

fluxctl list-images -w dev:deployment/vehement-bunny-extractors

fluxctl list-images -w dev:helmrelease/extractors

2opremio commented Jan 8, 2020

stefanprodan commented Jan 8, 2020

alastairs commented Jan 17, 2020

`HelmRelease`

`fluxctl list-workloads -ndev`

`fluxctl list-images -w dev:deployment/vehement-bunny-extractors`

`fluxctl list-images -w dev:helmrelease/extractors`