Support Installation Specification #1684

carolynvs · 2021-07-25T12:35:50Z

What does this change

Move all documents and data access code into Porter
Funnel most access to cnab-go/claims through porter's own pkg/claims (soon to be renamed to pkg/installations). I was seeing bugs and compilation errors because it's so easy to confuse the two or accidentally use the struct from the wrong package. This makes sure that all cnab installation data stuff goes through Porter.
Base storage plugin protocol on mongodb
Remove BackingStore and all autoconnect logic. We now connect to the storage plugin just in time, connecting as needed and holding the connection until porter exits. This means that tests that access storage must call Teardown() to release the connection.
Change filesystem storage plugin to run an instance of mongo and write the db files to PORTER_HOME.
Add namespace to all documents
Support namespaces, e.g. porter list -n dev, PORTER_NAMESPACE and namespace config
Run internal plugins inside the porter process, instead of execing back into itself. Yes this means that internal plugins are treated a bit differently. The protocol is still the same though and it greatly improves debugging.

TODO FOR THIS PR

Rename "filesystem" plugin to "docker-mongodb" (in this PR)
Document the mongodb-docker plugin (In this PR)

TODO AFTER THIS PR

Switch from using docker CLI directly, to using the docker library (in follow-up PR)
Add labels to all documents (in follow-up PR)
Add --all-namespaces, which defaults --namespace to "*" appropriately (in folow-up PR)

What issue does it fix

Fixes #1177
First part of #1683

Notes for the reviewer

Sorry it's giant, I needed to make sure the entire thing worked end-to-end. It may work better to do the review in a couple screen sharing sessions.

Checklist

Unit Tests
Documentation
Schema (porter.yaml)

carolynvs · 2021-07-30T20:04:10Z

/azp run porter-integration

azure-pipelines · 2021-07-30T20:04:21Z

Azure Pipelines successfully started running 1 pipeline(s).

carolynvs · 2021-08-03T13:18:08Z

/azp run porter-integration

azure-pipelines · 2021-08-03T13:18:19Z

Azure Pipelines successfully started running 1 pipeline(s).

* Change storage protocol to mimic mongo. * Switch the filesystem plugin to use a mongo instance in a container. * Add a mongo plugin. * Add installation document and rename claim -> run. * Move all storage document formats and data access into porter. Signed-off-by: Carolyn Van Slyck <[email protected]>

Signed-off-by: Carolyn Van Slyck <[email protected]>

If you are still using the filesystem plugin you will see the following error: Error: could not list installations: could not load storage plugin: unsupported internal storage plugin specified storage.porter.filesystem Signed-off-by: Carolyn Van Slyck <[email protected]>

carolynvs · 2021-08-05T18:52:38Z

/azp run porter-integration

azure-pipelines · 2021-08-05T18:52:47Z

Azure Pipelines successfully started running 1 pipeline(s).

vdice

👏 Impressive work, as always! These changes set Porter up to be far more performant and efficient. Majority of comments are minor and usually just typo-related.

It looks like there may be one more TODO in the description above, around documenting the mongodb-docker plugin -- does that still need to be added?

vdice · 2021-08-06T14:39:16Z

pkg/claims/installation.go

+		i.Status.InstallationCompleted = true
+	}
+
+	// TODO(carolynvs): set this before running the bundle, it should trigger the execution


I can't recall -- should this be addressed now or can defer to a follow-up?

That is part of #1704

pkg/claims/output.go

pkg/claims/provider.go

pkg/claims/run.go

pkg/secrets/pluginstore/store.go

pkg/storage/migrations/helpers.go

pkg/storage/plugin_adapter.go

pkg/storage/plugins/mongodb/mongodb.go

pkg/storage/pluginstore/store.go

carolynvs · 2021-08-06T19:49:00Z

Oh! Yes you are right, I wanted to document the plugin first before merging this PR. 👍

Our documents store PORTER'S schema for the document. If we need to convert that document to a CNAB representation, then we will stamp the converted document with the appropriate schema version for that spec. Some docs don't need it at all and I've removed it. Signed-off-by: Carolyn Van Slyck <[email protected]>

Signed-off-by: Carolyn Van Slyck <[email protected]>

carolynvs · 2021-08-06T21:03:34Z

I've fixed all the feedback I think. I'm going to leave the documentation for Monday and peace out for the weekend. Thanks for the review!

Signed-off-by: Carolyn Van Slyck <[email protected]>

carolynvs · 2021-08-09T15:17:22Z

https://deploy-preview-1684--porter.netlify.app/plugins/
https://deploy-preview-1684--porter.netlify.app/plugins/mongodb/
https://deploy-preview-1684--porter.netlify.app/plugins/mongodb-docker/

vdice

One minor docs Q, otherwise LGTM!

vdice · 2021-08-09T16:32:06Z

docs/content/plugins/mongodb-docker.md

+## Plugin Configuration
+
+To use the mongodb plugin, add the following config to porter's config file
+(default location: `~/.porter/config.toml`). Replace `conn_str` with the


Replace conn_str with the connection string for your MongoDB server. should be removed, right?

oops! thanks

Signed-off-by: Carolyn Van Slyck <[email protected]>

carolynvs changed the title ~~Redo data access layer~~ Support Installation Specification Jul 25, 2021

carolynvs force-pushed the swap-storage branch 2 times, most recently from cdb648b to 4c2cf8f Compare July 30, 2021 14:12

carolynvs force-pushed the swap-storage branch from 0279647 to c920d16 Compare August 2, 2021 17:33

carolynvs force-pushed the swap-storage branch from 1d9dad2 to ca43aa5 Compare August 3, 2021 15:09

This was referenced Aug 5, 2021

Add command to show historical runs of porter #1699

Closed

Migrate data from v0.38 format to v1.0 #1700

Closed

carolynvs force-pushed the swap-storage branch from cf0641b to 5da0581 Compare August 5, 2021 14:58

carolynvs added 2 commits August 5, 2021 10:58

Fix stuff found after reading my own code...

15d7389

Signed-off-by: Carolyn Van Slyck <[email protected]>

carolynvs marked this pull request as ready for review August 5, 2021 18:55

carolynvs requested review from jeremyrickard and vdice as code owners August 5, 2021 18:55

carolynvs mentioned this pull request Aug 6, 2021

Support labels #1701

Merged

3 tasks

vdice reviewed Aug 6, 2021

View reviewed changes

carolynvs added 6 commits August 6, 2021 15:25

Remove commented out code and document claimstore interface

fd8f59e

Signed-off-by: Carolyn Van Slyck <[email protected]>

Clarify debug statement for the current namespace

e53fb36

Signed-off-by: Carolyn Van Slyck <[email protected]>

Fix copy/paste error

10fe12b

Signed-off-by: Carolyn Van Slyck <[email protected]>

Fix typos

d2829a7

Signed-off-by: Carolyn Van Slyck <[email protected]>

Fix names for plugins in our config file

a122df7

Signed-off-by: Carolyn Van Slyck <[email protected]>

carolynvs added 3 commits August 9, 2021 10:06

Document built-in plugins

f0afe85

Signed-off-by: Carolyn Van Slyck <[email protected]>

Fix padding for list pages

6ddc321

Signed-off-by: Carolyn Van Slyck <[email protected]>

Fix formatting

9e2695d

Signed-off-by: Carolyn Van Slyck <[email protected]>

vdice approved these changes Aug 9, 2021

View reviewed changes

Fix exmaple for mongodb-docker

6e9a4ec

Signed-off-by: Carolyn Van Slyck <[email protected]>

carolynvs merged commit dae3283 into getporter:release/v1 Aug 9, 2021

carolynvs deleted the swap-storage branch August 9, 2021 17:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Installation Specification #1684

Support Installation Specification #1684

carolynvs commented Jul 25, 2021 •

edited

Loading

carolynvs commented Jul 30, 2021

azure-pipelines bot commented Jul 30, 2021

carolynvs commented Aug 3, 2021

azure-pipelines bot commented Aug 3, 2021

carolynvs commented Aug 5, 2021

azure-pipelines bot commented Aug 5, 2021

vdice left a comment

vdice Aug 6, 2021

carolynvs Aug 6, 2021

carolynvs commented Aug 6, 2021

carolynvs commented Aug 6, 2021

carolynvs commented Aug 9, 2021

vdice left a comment

vdice Aug 9, 2021

carolynvs Aug 9, 2021

Support Installation Specification #1684

Support Installation Specification #1684

Conversation

carolynvs commented Jul 25, 2021 • edited Loading

What does this change

What issue does it fix

Notes for the reviewer

Checklist

carolynvs commented Jul 30, 2021

azure-pipelines bot commented Jul 30, 2021

carolynvs commented Aug 3, 2021

azure-pipelines bot commented Aug 3, 2021

carolynvs commented Aug 5, 2021

azure-pipelines bot commented Aug 5, 2021

vdice left a comment

Choose a reason for hiding this comment

vdice Aug 6, 2021

Choose a reason for hiding this comment

carolynvs Aug 6, 2021

Choose a reason for hiding this comment

carolynvs commented Aug 6, 2021

carolynvs commented Aug 6, 2021

carolynvs commented Aug 9, 2021

vdice left a comment

Choose a reason for hiding this comment

vdice Aug 9, 2021

Choose a reason for hiding this comment

carolynvs Aug 9, 2021

Choose a reason for hiding this comment

carolynvs commented Jul 25, 2021 •

edited

Loading