Skip to content
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.

Add Manifest Factorization through .flux.yaml config files #1848

Merged
merged 11 commits into from
May 30, 2019

Conversation

2opremio
Copy link
Contributor

@2opremio 2opremio commented Mar 20, 2019

Want to test this? Go to https://github.com/weaveworks/flux-kustomize-example and follow the instructions.

This feature (behind the flag --manifest-generation) is now available in our prereleases repo: https://hub.docker.com/r/weaveworks/flux-prerelease/tags

This PR implements Generators and Updaters as specified in Generalizing Manifest Factorization in Weave Flux.

It is meant as a Proof of Concept and will only be merged if/when there is positive feedback.

One major wart is that Kubernetes doesn't offer Ephemeral containers yet. It's been in the works for quite a while ( kubernetes/kubernetes#59416 ) but until then there is no simple and (reasonably) secure way to run generators and releasers. For now, we will simply assume that the command executables must live in the flux container.

  • Abstract out manifest file operations (done by using an adapter of Manifests: ResourceStore)
  • Implement Generators and Updaters
  • Add the most popular factorization tools to the Flux container image (only Kustomize for now)
  • Create a comprehensive example (https://github.com/2opremio/flux-kustomize-demo)
  • Address FIXMEs TODOs and performance issues
  • Hide behind an experimental flag
  • Add more unit tests
  • Circulate concept through the community to gather feedback

Fixes #943 #1261 #1811 (and #1420 to certain extend)

@2opremio 2opremio force-pushed the generators-releasers branch from df0cd55 to 977a866 Compare March 20, 2019 23:39
@2opremio 2opremio force-pushed the generators-releasers branch from 977a866 to f5b31c2 Compare March 29, 2019 14:10
@2opremio 2opremio changed the title Proof of Concept: Generators and Releasers Proof of Concept: Generators and Updaters Mar 29, 2019
@2opremio 2opremio force-pushed the generators-releasers branch 5 times, most recently from 3fc3952 to 7f54431 Compare April 3, 2019 19:23
cluster/manifests.go Outdated Show resolved Hide resolved
@2opremio 2opremio force-pushed the generators-releasers branch 12 times, most recently from c83d6de to 15dad2f Compare April 11, 2019 11:31
@2opremio 2opremio force-pushed the generators-releasers branch 5 times, most recently from 5cf9625 to db644b0 Compare April 11, 2019 18:04
cmd/fluxd/main.go Outdated Show resolved Hide resolved
@2opremio 2opremio marked this pull request as ready for review April 11, 2019 18:39
Copy link
Member

@squaremo squaremo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting into more detailed stuff now. I don't think there are any show stoppers -- responses arguing that things are invalid, can be ignored, or can be deferred would suffice at this point.

My primary concern is that this will produce exactly the same results as before when --manifest-generation is off, and I'm fairly well persuaded that will be the case. The rest is negotiable :-)

cluster/kubernetes/manifests.go Outdated Show resolved Hide resolved
git/working.go Outdated Show resolved Hide resolved
git/working.go Outdated Show resolved Hide resolved
release/context.go Outdated Show resolved Hide resolved
release/releaser_test.go Outdated Show resolved Hide resolved
resourcestore/configfile.go Show resolved Hide resolved
resourcestore/fileresourcestore.go Outdated Show resolved Hide resolved
resourcestore/fileresourcestore.go Show resolved Hide resolved
resourcestore/fileresourcestore.go Show resolved Hide resolved
site/fluxyaml-config-files.md Outdated Show resolved Hide resolved
@2opremio 2opremio force-pushed the generators-releasers branch 5 times, most recently from fd0e871 to d11ea84 Compare May 29, 2019 11:30
@2opremio 2opremio force-pushed the generators-releasers branch 3 times, most recently from eeb9cf4 to 8d27f07 Compare May 29, 2019 16:25
@2opremio
Copy link
Contributor Author

2opremio commented May 29, 2019

Post-merge checklist:

@2opremio 2opremio force-pushed the generators-releasers branch from 8d27f07 to 2258bf7 Compare May 29, 2019 18:13
2opremio and others added 11 commits May 30, 2019 13:38
* Remove dependency on `git.Checkout` and rename `checkoutManager` to
  `fileResourceStore`
* Inline configfilemanager and manifestfilemanager into the new
  `fileResourceStore`
PolicyTranslator exists because the `.flux.yaml` files give commands
for updating annotations, rather than updating policies, following on
from how KubeYAML was used. But the fact that policies appear as
annotations in Kubernetes resources is really up to the code in
cluster/kubernetes.

This commit changes the format of `.flux.yaml` files so a command for
updating _policies_ is supplied, rather than for annotations. All the
translation of policies to annotations is then done in
cluster/kubernetes.

There still needs to be a little code to evaluate the qualified policy
updates (i.e., tag_all) against a particular workload, but this can be
done without reference to whether they end up as annotations.
Also, fix git file adding (the CommitAndPush check doesn't detect untracked files).
@squaremo squaremo force-pushed the generators-releasers branch from 2258bf7 to b55d0ff Compare May 30, 2019 12:38
Copy link
Member

@squaremo squaremo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a substantial amount of work Fons. Well done on the thoughtful restructuring of the code, and pushing the PR through reviews, pre-releases, and other trials.

As a last smoke test, I tried the image built from this branch locally, both with the flux-example repo (and --manifest-generation=false) and the repo flux-kustomize-demo (--manifest-generation=true).

OK, let's merge this and let it soak in master ..

@squaremo squaremo merged commit e81b697 into fluxcd:master May 30, 2019
@2opremio 2opremio deleted the generators-releasers branch May 30, 2019 15:34
@2opremio
Copy link
Contributor Author

Thanks @squaremo !!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for ksonnet
3 participants