inconsistent repository metadata: missing metadata for image tag

[ikaruswill]

Describe the bug

Flux is unable to automate workloads with multi-arch images with manifests that do not have linux/amd64.

Logs show:

inconsistent repository metadata: missing metadata for image tag \"0.1.3\"

fluxctl list-images on a workload that has uses 0.1.3 of the image gets:

drone:cronjob/tag-latest-release-home-assistant-ldap  tag-latest-release
                                                                          '->
                                                                              0.1.3-linux-arm64         21 May 20 09:27 UTC
                                                                              0.1.3-linux-arm           21 May 20 09:28 UTC
                                                                          '->
                                                                          '->
                                                                          '->
                                                                          '->
                                                                              0.1-linux-arm64           21 May 20 09:27 UTC
                                                                              0.1-linux-arm             21 May 20 09:28 UTC

To Reproduce

Steps to reproduce the behaviour:

1. Deploy flux with stable helm chart
2. Build and push a multi-arch image 0.1.0 with a manifest linking to linux/arm/v7 and linux/arm64/v8 to Docker Hub
3. Create a deployment with the multi-arch image on an linux/arm or linux/arm64 node
4. Set up workload automation with semver:~0
5. Build and push the same multi-arch image with tag 0.1.1
6. Follow flux logs and see that the workload will not update the container image to 0.1.1

Expected behavior

Workload automation should update container images that do not have linux/amd64 in the manifest.

Logs

On deleting and recreating flux and memcached, one would observe this:

ts=2020-05-21T10:36:13.121991999Z caller=repocachemanager.go:267 component=warmer canonical_name=index.docker.io/ikaruswill/tag-latest-release auth={map[]} excluded="no suitable manifest (linux amd64) in manifestlist" ref=ikaruswill/tag-latest-release:0.1.3

Followed by:

ts=2020-05-21T13:36:36.185357313Z caller=images.go:95 component=sync-loop workload=drone:cronjob/tag-latest-release-home-assistant-ldap container=tag-latest-release repo=ikaruswill/tag-latest-release pattern=semver:~0 current=ikaruswill/tag-latest-release:0.1.3 warning="inconsistent repository metadata: missing metadata for image tag \"0.1.3\"" action="skip container"

Additional context

• Flux version: 1.19.0
• Kubernetes version: 1.17.4
• Git provider: Github
• Container registry provider: Docker hub

[ikaruswill]

I've checked pkg/registry/client.go and found this:

case *manifestlist.DeserializedManifestList:
	var list manifestlist.ManifestList = deserialised.ManifestList
	// TODO(michael): is it valid to just pick the first one that matches?
	for _, m := range list.Manifests {
		if m.Platform.OS == "linux" && m.Platform.Architecture == "amd64" {
			manifest, fetchErr = manifests.Get(ctx, m.Digest, digestOpt)
			goto interpret
		}
	}
	entry := ImageEntry{}
	entry.ExcludedReason = "no suitable manifest (linux amd64) in manifestlist"
	return entry, nil

I'm not versed in Go, but based on what I see, if there are no manifests with linux/amd64, an empty ImageEntry with no Info will be returned with a nil value for error

This would trigger the error in pkg/image/image.go:

func (rm RepositoryMetadata) GetImageTagInfo() ([]Info, error) {
	result := make([]Info, len(rm.Tags), len(rm.Tags))
	for i, tag := range rm.Tags {
		info, ok := rm.Images[tag]
		if !ok {
			return nil, fmt.Errorf("missing metadata for image tag %q", tag)
		}
		result[i] = info
	}
	return result, nil
}

Correct me if I'm wrong but does this mean that for an image manifest to be processed/cached correctly, flux needs the presence of a linux/amd64 entry?

IMO, a valid manifest should not need linux/amd64 but should depend on the os/architecture that the workload runs on.

There have been some community efforts to build flux for other architectures, most notably, arm and arm64 such as #1761, so I see no danger in processing images that aren't the same as the official flux image.

[kingdonb]

Since #3141 is closed, I will close this too.

Thanks for your contribution and thanks for using Flux!