WIP: Improvement/956 saltify solution lifecycle #1487

sayf-eddine-scality · 2019-07-30T14:56:24Z

Component:
solutions, salt

Context:
Solutions ISOs need a salt runners to check and import them into the cluster
Summary:
Salt runners all batteries included has been added with a bash script to automate these actions
Acceptance criteria:

Closes: #956

bert-e · 2019-07-30T14:56:26Z

Hello sayf-eddine-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2019-07-30T14:56:31Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

TeddyAndrieux

IMHO the approach here is not good as it's really strange to maintain a configmap when we configure solution.
For me the configmap should contain a list of solutions and salt should only read it (from an ext pillar) and configure/unconfigure the ISOs to match what is in the configmap. And if needed maybe the script can edit the configmap or something in the UI but for me the salt states should only apply what is in configmap and not maintain it.

But it's only my opinion :)

salt/_modules/solutions.py

salt/metalk8s/solutions/configured.sls

bert-e · 2019-07-30T15:37:46Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@TeddyAndrieux

sayf-eddine-scality · 2019-07-31T06:15:43Z

IMHO the approach here is not good as it's really strange to maintain a configmap when we configure solution.
For me the configmap should contain a list of solutions and salt should only read it (from an ext pillar) and configure/unconfigure the ISOs to match what is in the configmap. And if needed maybe the script can edit the configmap or something in the UI but for me the salt states should only apply what is in configmap and not maintain it.

But it's only my opinion
@gdemonet what do you think ?

gdemonet · 2019-07-31T06:42:17Z

IMHO the approach here is not good as it's really strange to maintain a configmap when we configure solution.
For me the configmap should contain a list of solutions and salt should only read it (from an ext pillar) and configure/unconfigure the ISOs to match what is in the configmap. And if needed maybe the script can edit the configmap or something in the UI but for me the salt states should only apply what is in configmap and not maintain it.

But it's only my opinion :)

Mmh we designed it to be some kind of state persistence to be read/written by Salt when managing Solutions. The declarative listing comes from a file, /etc/metalk8s/solutions.yaml, and the ConfigMap serves as an observation of what is really deployed (even if it can be tampered with, granted). We need it for two reasons:

comparing the desired Solution versions (from the file) with what is deployed (from the CM)
list deployed Solution versions from the UI

gdemonet

Some design issues: I think we don't want the ext_pillar to compute anything, only forward data sources (a config file, or the ConfigMap) to the minions.

Also, the format of solutions.yaml needs to be reviewed - it should look like:

kind: SolutionsConfiguration
apiVersion: v1alpha1
archives:
  - /archives/hyperdrive-1.0.0.iso
  - /archives/hyperdrive-1.1.0.iso
  - /zenko/archives/zenko.iso

Our tooling should behave roughly like we do for MetalK8s ISOs: first read config file to list all ISOs we are concerned with, then read product info from the ISO without mounting it, then mount the ISO, finally configure the repositories to include the mounted ISO.

The ConfigMap doesn't need to exist for this functionality - however, it serves two other purposes:

it is read from the Platform UI to know about deployed Solutions
it is read by the metalk8s.orchestrate.solutions runner to know which Solution versions need to be removed

Otherwise, that's great progress, thanks!

buildchain/buildchain/salt_tree.py

salt/_modules/solutions.py

salt/metalk8s/solutions/mounted.sls

scripts/solution-manager.sh

gdemonet · 2019-07-31T09:51:32Z

scripts/solution-manager.sh

+        echo "solution path name and version are required"
+        exit 1
+    fi
+    echo "Updating metalk8s.solutions.yaml"


Suggested change

echo "Updating metalk8s.solutions.yaml"

echo "Updating $SOLUTION_CONFIG"

scripts/solution-manager.sh

bert-e · 2019-07-31T11:44:10Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

salt/_modules/solutions.py

salt/metalk8s/orchestrate/solutions/configured.sls

bert-e · 2019-08-05T12:52:38Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

buildchain/buildchain/iso.py

bert-e · 2019-08-08T10:01:26Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

salt/_pillar/metalk8s_solutions.py

salt/metalk8s/orchestrate/solutions/init.sls

salt/metalk8s/repo/installed.sls

bert-e · 2019-08-12T07:50:44Z

Branches have diverged

This pull request's source branch improvement/956-saltify-solution-lifecycle has diverged from
development/2.4 by more than 50 commits.

To avoid any integration risks, please re-synchronize them using one of the
following solutions:

Merge origin/development/2.4 into improvement/956-saltify-solution-lifecycle
Rebase improvement/956-saltify-solution-lifecycle onto origin/development/2.4

Note: If you choose to rebase, you may have to ask me to rebuild
integration branches using the reset command.

bert-e · 2019-08-19T14:54:26Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

salt/metalk8s/orchestrate/solutions/init.sls

salt/metalk8s/solutions/mounted.sls

TeddyAndrieux

LGTM

bert-e · 2019-08-29T08:38:34Z

Build failed

The build for commit did not succeed in branch improvement/956-saltify-solution-lifecycle.

The following options are set: approve

bert-e · 2019-08-29T09:12:24Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following options are set: approve

TeddyAndrieux · 2019-08-29T09:37:09Z

salt/_modules/metalk8s_solutions.py

+        with open(SOLUTIONS_CONFIG_FILE, 'r') as fd:
+            content = yaml.safe_load(fd)
+    except Exception as exc:
+        log.exception('Failed to load "%s": %s',


Do not log.exception in module that run in ext_pillar please.

For me in module we should never log.exception we should only raise the exception and the caller handle the exception like he want

bert-e · 2019-08-29T09:37:18Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@TeddyAndrieux

The following options are set: approve

Fixes: #956

bert-e · 2019-08-29T10:37:14Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.4

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2019-08-29T11:19:29Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.4

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

Please check the status of the associated issue None.

Goodbye sayf-eddine-scality.

sayf-eddine-scality added moonshot topic:solutions Anything related to external "Solutions" integration with the platform labels Jul 30, 2019

sayf-eddine-scality added this to the MetalK8s 2.4.0-alpha1 milestone Jul 30, 2019

sayf-eddine-scality requested review from gdemonet and carlito-scality July 30, 2019 14:56

TeddyAndrieux requested changes Jul 30, 2019

View reviewed changes

salt/_modules/solutions.py Outdated Show resolved Hide resolved

salt/_modules/solutions.py Outdated Show resolved Hide resolved

salt/_modules/solutions.py Outdated Show resolved Hide resolved

salt/metalk8s/solutions/configured.sls Outdated Show resolved Hide resolved

gdemonet suggested changes Jul 31, 2019

View reviewed changes

NicolasT reviewed Aug 1, 2019

View reviewed changes

salt/_modules/solutions.py Outdated Show resolved Hide resolved

NicolasT reviewed Aug 1, 2019

View reviewed changes

salt/metalk8s/orchestrate/solutions/configured.sls Outdated Show resolved Hide resolved

sayf-eddine-scality force-pushed the improvement/956-saltify-solution-lifecycle branch from 5efa81b to 1243b75 Compare August 5, 2019 12:52

slaperche-scality reviewed Aug 5, 2019

View reviewed changes

buildchain/buildchain/iso.py Outdated Show resolved Hide resolved

sayf-eddine-scality force-pushed the improvement/956-saltify-solution-lifecycle branch from 02f4e4f to 31c5d65 Compare August 7, 2019 08:56

gdemonet reviewed Aug 8, 2019

View reviewed changes

salt/_pillar/metalk8s_solutions.py Outdated Show resolved Hide resolved

gdemonet reviewed Aug 8, 2019

View reviewed changes

salt/metalk8s/orchestrate/solutions/init.sls Outdated Show resolved Hide resolved

gdemonet reviewed Aug 8, 2019

View reviewed changes

salt/metalk8s/orchestrate/solutions/init.sls Outdated Show resolved Hide resolved

gdemonet reviewed Aug 8, 2019

View reviewed changes

salt/metalk8s/repo/installed.sls Outdated Show resolved Hide resolved

sayf-eddine-scality force-pushed the improvement/956-saltify-solution-lifecycle branch from 66fdb5c to f681b2a Compare August 12, 2019 13:16

sayf-eddine-scality force-pushed the improvement/956-saltify-solution-lifecycle branch from 5f7c678 to 04ee105 Compare August 19, 2019 14:54

sayf-eddine-scality force-pushed the improvement/956-saltify-solution-lifecycle branch from 630e1fc to c885d80 Compare August 29, 2019 07:28

TeddyAndrieux reviewed Aug 29, 2019

View reviewed changes

salt/metalk8s/orchestrate/solutions/init.sls Outdated Show resolved Hide resolved

TeddyAndrieux reviewed Aug 29, 2019

View reviewed changes

salt/metalk8s/solutions/mounted.sls Outdated Show resolved Hide resolved

sayf-eddine-scality force-pushed the improvement/956-saltify-solution-lifecycle branch 2 times, most recently from 3664a01 to f380c31 Compare August 29, 2019 08:24

TeddyAndrieux previously approved these changes Aug 29, 2019

View reviewed changes

sayf-eddine-scality dismissed TeddyAndrieux’s stale review via ab27210 August 29, 2019 09:12

TeddyAndrieux approved these changes Aug 29, 2019

View reviewed changes

TeddyAndrieux requested changes Aug 29, 2019

View reviewed changes

sayf-eddine-scality added 6 commits August 29, 2019 11:53

Clean configmap if last version is removed

94ec76e

Fixes: #956

Various code style fixes

b79cb8f

Fix: remove admin-uis configmap

1fa20c4

Undeploy solution objects if last version is removed

df24d1b

Add registry ip:port to ui-branding configmap

42bb608

Create solutions config file if it does not exist

f540620

sayf-eddine-scality force-pushed the improvement/956-saltify-solution-lifecycle branch from ab27210 to f540620 Compare August 29, 2019 09:54

TeddyAndrieux approved these changes Aug 29, 2019

View reviewed changes

bert-e merged commit f540620 into development/2.4 Aug 29, 2019

bert-e deleted the improvement/956-saltify-solution-lifecycle branch August 29, 2019 11:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Improvement/956 saltify solution lifecycle #1487

WIP: Improvement/956 saltify solution lifecycle #1487

sayf-eddine-scality commented Jul 30, 2019

bert-e commented Jul 30, 2019

bert-e commented Jul 30, 2019

TeddyAndrieux left a comment •

edited

Loading

bert-e commented Jul 30, 2019

sayf-eddine-scality commented Jul 31, 2019

gdemonet commented Jul 31, 2019

gdemonet left a comment

gdemonet Jul 31, 2019

bert-e commented Jul 31, 2019

bert-e commented Aug 5, 2019

bert-e commented Aug 8, 2019

bert-e commented Aug 12, 2019

bert-e commented Aug 19, 2019

TeddyAndrieux left a comment

bert-e commented Aug 29, 2019

bert-e commented Aug 29, 2019

TeddyAndrieux Aug 29, 2019 •

edited

Loading

bert-e commented Aug 29, 2019

bert-e commented Aug 29, 2019

bert-e commented Aug 29, 2019

	echo "Updating metalk8s.solutions.yaml"
	echo "Updating $SOLUTION_CONFIG"

WIP: Improvement/956 saltify solution lifecycle #1487

WIP: Improvement/956 saltify solution lifecycle #1487

Conversation

sayf-eddine-scality commented Jul 30, 2019

bert-e commented Jul 30, 2019

Hello sayf-eddine-scality,

bert-e commented Jul 30, 2019

Waiting for approval

TeddyAndrieux left a comment • edited Loading

Choose a reason for hiding this comment

bert-e commented Jul 30, 2019

Waiting for approval

sayf-eddine-scality commented Jul 31, 2019

gdemonet commented Jul 31, 2019

gdemonet left a comment

Choose a reason for hiding this comment

gdemonet Jul 31, 2019

Choose a reason for hiding this comment

bert-e commented Jul 31, 2019

Waiting for approval

bert-e commented Aug 5, 2019

Waiting for approval

bert-e commented Aug 8, 2019

Waiting for approval

bert-e commented Aug 12, 2019

Branches have diverged

bert-e commented Aug 19, 2019

Waiting for approval

TeddyAndrieux left a comment

Choose a reason for hiding this comment

bert-e commented Aug 29, 2019

Build failed

bert-e commented Aug 29, 2019

Waiting for approval

TeddyAndrieux Aug 29, 2019 • edited Loading

Choose a reason for hiding this comment

bert-e commented Aug 29, 2019

Waiting for approval

bert-e commented Aug 29, 2019

In the queue

bert-e commented Aug 29, 2019

TeddyAndrieux left a comment •

edited

Loading

TeddyAndrieux Aug 29, 2019 •

edited

Loading