Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add templating to porter config file #1879

Merged
merged 2 commits into from
Feb 1, 2022
+456 −138
Merged

Add templating to porter config file #1879

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dfeb0e7
Add templating to porter config file
carolynvs Jan 28, 2022
ab56dbb
Review feedback
carolynvs Feb 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
/docs/public/
/docs/content/operator
/bin
/porter
.DS_Store
Expand Down
8 changes: 7 additions & 1 deletion cmd/porter/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,12 @@ var usageText string

func main() {
run := func() int {
ctx := context.Background()
p := porter.New()
if err := p.Connect(ctx); err != nil {
fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}

rootCmd := buildRootCommandFrom(p)

Expand Down Expand Up @@ -116,8 +121,9 @@ Try our QuickStart https://porter.sh/quickstart to learn how to use Porter.
case "porter", "help", "version", "docs":
return nil
default:
// Reload configuration with the now parsed cli flags
p.DataLoader = cli.LoadHierarchicalConfig(cmd)
err := p.LoadData()
err := p.Connect(cmd.Context())
if err != nil {
return err
}
Expand Down
272 changes: 201 additions & 71 deletions docs/content/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,99 +3,171 @@ title: Configuration
description: Controlling Porter with its config file, environment variables and flags
---

Porter has a hierarchical configuration system that loads configuration values in the following precedence order:

* Flags (highest)
* Environment Variables
* Config File (lowest)

You may set a default value for a configuration value in the config file, override it with an environment variable, and then override both for a particular command with a flag.

* [Flags](#flags)
* [Environment Variables](#environment-variables)
* [Config File](#config-file)
* [Experimental Feature Flags](#experimental-feature-flags)
* [Build Drivers](#build-drivers)
* [Structured Logs](#structured-logs)
* [Common Configuration Settings](#common-configuration-settings)
* [Set Current Namespace](#namespace)
* [Enable Debug Output](#debug)
* [Debug Plugins](#debug-plugins)
* [Output Formatting](#output)
* [Allow Docker Host Access](#allow-docker-host-access)

Porter's configuration system has a precedence order:
## Flags

* Flags (highest)
* Environment Variables
* Config File (lowest)
Nearly all of Porter's configuration, except global configuration such as secret accounts, storage accounts, or telemetry, are configurable by flags.
Use the `porter help` command to view available flags.

You may set a default value for a configuration value in the config file,
override it in a shell session with an environment variable and then override
both in a particular command with a flag.
## Environment Variables

* [Set Current Namespace](#namespace)
* [Enable Debug Output](#debug)
* [Debug Plugins](#debug-plugins)
* [Output Formatting](#output)
* [Allow Docker Host Access](#allow-docker-host-access)
Flags have corresponding environment variables that you can use so that you don't need to manually set the flag every time.
The flag will default to the value of the environment variable, when defined.
Global configuration settings can also be specified with an environment variable.
For example, the experimental config file setting maps to PORTER_EXPERIMENTAL, and accepts a comma-separated list of values.

## Flags
\--flag maps to the environment variable of PORTER_FLAG.
Dashes in the flag name are represented as underscores in the environment variable name.
So \--another-flag maps to the environment variable PORTER_ANOTHER_FLAG

### Namespace
`--namespace` specifies the current namespace.
For example, you can set PORTER_OUTPUT=json and then all subsequent porter commands will act as though the \--output=json flag was passed.

### Debug
## Config File

`--debug` is a flag that is understood not only by the porter client but also the
runtime and most mixins. They may use it to print additional information that
may be useful when you think you may have found a bug, when you want to know
what commands they are executing, or when you need really verbose output to send
to the developers.
Porter's configuration file is located in the PORTER_HOME directory, by default ~/.porter/.
The file name should be config.FILE_EXTENSION, where the file extension can be json, toml, yaml, or hcl.
For example, If you defined the configuration in YAML, the file is named config.yaml.

### Debug Plugins
Do not embed sensitive data in the configuration file.
Instead, use templates to inject environment variables or secrets in the configuration file.
Environment variables are specified with ${env.NAME}, where name is case-sensitive.
Secrets are specified with ${secret.KEY} and case sensitivity depends upon the secrets plugin used.

`--debug-plugins` controls if logs related to communication
between porter and its plugins should be printed when debugging. This can be _very_
verbose, so it is not turned on by default when debug is true.
Below is an example configuration file in TOML:

### Output
```toml
# ~/.porter/config.toml

`--output` controls the format of the output printed by porter. Each command
supports a different set of allowed outputs though usually there is some
combination of: `table`, `json` and `yaml`.
# Set the default namespace
namespace = "dev"

### Allow Docker Host Access
# Include debug logs
debug = true

`--allow-docker-host-access` controls whether the local Docker daemon
should be made available to executing bundles. This flag is available for the
following commands: [install], [upgrade], [invoke] and [uninstall]. When this
value is set to true, bundles are executed in a privileged container with the
docker socket mounted. This allows you to use Docker from within your bundle,
such as `docker push`, `docker-compose`, or docker-in-docker.
# Include debug logs from the plugins
debug-plugins = true

🚨 **There are security implications to enabling access! You should trust any
bundles that you execute with this setting enabled as it gives them elevated
access to the host machine.**
# Default command output to JSON
output = "json"

⚠️️ This configuration setting is only available when you are in an environment
that provides access to the local docker daemon. Therefore it does not work with
the Azure Cloud Shell driver.
# Allow all bundles access to the Docker Host
allow-docker-host-access = true

## Environment Variables
# Enable experimental v1 features
experimental = ["build-drivers", "structured-logs"]

Flags have corresponding environment variables that you can use so that you
don't need to manually set the flag every time. The flag will default to the
value of the environment variable, when defined.
# Use Docker buildkit to build the bundle
build-driver = "buildkit"

`--flag` has a corresponding environment variable of `PORTER_FLAG` and `--another-flag`
corresponds to the environment variable `PORTER_ANOTHER_FLAG`.
# Use the storage configuration named devdb
default-storage = "devdb"

# When default-storage is not set, use the mongodb-docker plugin.
# This mode does not support additional configuration for the plugin.
# If the plugin requires configuration, use default-storage and define
# the configuration in the storage section.
default-storage-plugin = "mongodb-docker"

# Use the secrets configuration named mysecrets
default-secrets = "mysecrets"

# When default-secrets is not set, use the kubernetes.secret plugin.
# This mode does not support additional configuration for the plugin.
# If the plugin requires configuration, use default-secrets and define
# the configuration in the secrets section.
default-secrets-plugin = "kubernetes.secret"

# Defines storage accounts
[[storage]]
# The storage account name
name = "devdb"

# The plugin used to access the storage account
plugin = "mongodb"

# Additional configuration for storage account
# These values vary depending on the plugin used
[storage.config]
# The mongodb connection string
url = "${secret.porter-db-connection-string}"

# Timeout for database queries
timeout = 300

# Define secret store accounts
[[secrets]]
# The secret store name
name = "mysecrets"

# The plugin used to access the secret store account
plugin = "azure.keyvault"

# Additional configuration for secret store account
# These values vary depending on the plugin used
[secrets.config]
# The name of the secret vault
vault = "topsecret"

# The subscription where the vault is defined
subscription-id = "${env.AZURE_SUBSCRIPTION_ID}"

# Log command output to a file in PORTER_HOME/logs/
[logs]
# Log command output to a file
enabled = true

For example, you can set `PORTER_DEBUG=true` and then all subsequent porter
commands will act as though the `--debug` flag was passed.
# Sets the log level for what is written to the file
# Allowed values: debug, info, warn, error
level = "info"

## Config File
# Send trace and log data to an Open Telemetry collector
[telemetry]
# Enable trace collection
enabled = true

Common settings can be defaulted in the config file. The config file is located in
the PORTER_HOME directory (**~/.porter**), is named **config** and can be in any
of the following file types: JSON, TOML, YAML, HCL, envfile and Java Properties
files.
# Send telemetry via the grpc protocol
# Allowed values: http/protobuf, grpc
protocol = "grpc"

Below is an example configuration file in TOML
# The Open Telemetry collector endpoint
endpoint = "127.0.0.1:4318"

**~/.porter/config.toml**
```toml
namespace = "dev"
debug = true
debug-plugins = true
output = "json"
allow-docker-host-access = true
# Specify if the collector endpoint is secured with TLS
insecure = true

# Specify a certificate to connect to the collector endpoint
certificate = "/home/me/some-cert.pem"

# The compression type used when communicating with the collector endpoint
compression = "gzip"

# The timeout enforced when communicating with the collector endpoint
timeout = "3s"

# Additional headers to send to the open telemetry collector
[telemetry.headers]
environment = "dev"
owner = "myusername"
```

## Experimental Feature Flags
Expand All @@ -104,11 +176,11 @@ Porter sometimes uses feature flags to release new functionality for users to
evaluate, without affecting the stability of Porter. You can enable an experimental
feature by:

* Using the experimental global flag `--experimental flagA,flagB`.
* Using the experimental global flag \--experimental flagA,flagB.
The value is a comma-separated list of strings.
* Setting the PORTER_EXPERIMENTAL environment variable like so `PORTER_EXPERIMENTAL=flagA,flagB`.
* Setting the PORTER_EXPERIMENTAL environment variable like so PORTER_EXPERIMENTAL=flagA,flagB.
The value is a comma-separated list of strings.
* Setting the experimental field in the configuration file like so `experimental = ["flagA","flagB"]`.
* Setting the experimental field in the configuration file like so experimental = ["flagA","flagB"].
The value is an array of strings.

### Build Drivers
Expand All @@ -117,11 +189,11 @@ The **build-drivers** experimental feature flag enables using a different
driver to build OCI images used by the bundle, such as the installer.

You can set your desired driver with either using `porter build --driver`,
`PORTER_BUILD_DRIVER` environment variable, or in the configuration file with
`build-driver = "DRIVER"`
PORTER_BUILD_DRIVER environment variable, or in the configuration file with
build-driver = "DRIVER".

The default driver is [Docker], and the full list of available drivers
is below.
The default driver is docker, and the full list of available drivers
is below:

* **Docker**: Build an OCI image using the [Docker library], without buildkit support.
This requires access to a Docker daemon, either locally or remote.
Expand Down Expand Up @@ -212,4 +284,62 @@ experimental = ["structured-logs"]
owner = "me"
```

[otel]: https://github.com/open-telemetry/opentelemetry-specification/blob/v1.8.0/specification/protocol/exporter.md
[otel]: https://github.com/open-telemetry/opentelemetry-specification/blob/v1.8.0/specification/protocol/exporter.md

## Common Configuration Settings

Some configuration settings are applicable to many of Porter's commands and to save time you may want to set these values in the configuration file or with environment variables.

### Namespace
\--namespace specifies the current namespace.
It is set with the PORTER_NAMESPACE environment variable.

```toml
namespace = "dev"
```

### Debug

\--debug is a flag that is understood not only by the porter client but also the runtime and most mixins.
They may use it to print additional information that may be useful when you think you may have found a bug, when you want to know what commands they are executing, or when you need really verbose output to send
to the developers.
It is set with the PORTER_DEBUG environment variable.

```toml
debug = true
```

### Debug Plugins

\--debug-plugins controls if logs related to communication between porter and its plugins should be printed when debugging.
This can be _very_ verbose, so it is not turned on by default when debug is true.
It is set with the PORTER_DEBUG_PLUGINS environment variable.

```toml
debug-plugins = true
```

### Output

\--output controls the format of the command output printed by porter.
It is set with the PORTER_OUTPUT environment variable.
Each command supports a different set of allowed outputs though usually there is some combination of: plaintext, json, and yaml.

```toml
output = "json"
```

### Allow Docker Host Access

\--allow-docker-host-access controls whether the local Docker daemon or host should be made available to executing bundles.
It is set with the PORTER_ALLOW_DOCKER_HOST_ACCESS environment variable.

This flag is available for the following commands: install, upgrade, invoke, and uninstall.
When this value is set to true, bundles are executed in a privileged container with the docker socket mounted.
This allows you to use Docker from within your bundle, such as `docker push`, `docker-compose`, or docker-in-docker.

🚨 **There are security implications to enabling access!
You should trust any bundles that you execute with this setting enabled as it gives them elevated access to the host machine.**

⚠️️ This configuration setting is only available when you are in an environment that provides access to the local docker daemon.
Therefore, it does not work with the Azure Cloud Shell driver.
5 changes: 5 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ replace (
// go.mod doesn't propogate replacements in the dependency graph so I'm copying this from github.com/moby/buildkit
github.com/jaguilar/vt100 => github.com/tonistiigi/vt100 v0.0.0-20190402012908-ad4c4a574305

// expose-ast
// https://github.com/osteele/liquid/pull/59
github.com/osteele/liquid => github.com/carolynvs/liquid v1.2.5-0.20220131221838-2e107bef298f

// Fixes https://github.com/spf13/viper/issues/761
github.com/spf13/viper => github.com/getporter/viper v1.7.1-porter.2.0.20210514172839-3ea827168363
)
Expand Down Expand Up @@ -61,6 +65,7 @@ require (
github.com/moby/term v0.0.0-20201216013528-df9cb8a40635
github.com/olekukonko/tablewriter v0.0.4
github.com/opencontainers/go-digest v1.0.0
github.com/osteele/liquid v1.2.4
github.com/pelletier/go-toml v1.9.1
github.com/pivotal/image-relocation v0.0.0-20191111101224-e94aff6df06c
github.com/pkg/errors v0.9.1
Expand Down
4 changes: 4 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,8 @@ github.com/carolynvs/cnab-go v0.20.2-0.20210805155536-9a543e0636f4 h1:w6gndqIhqX
github.com/carolynvs/cnab-go v0.20.2-0.20210805155536-9a543e0636f4/go.mod h1:u/Y7piTNJuFs2KfQqmda6uqIx4rtqQ73H6IW7gudz7E=
github.com/carolynvs/datetime-printer v0.2.0 h1:Td3FU4YGzx0OogCMhCmLBTUTDPQcq0xlgCeMhAKZmMc=
github.com/carolynvs/datetime-printer v0.2.0/go.mod h1:p9W8ZUhmQUOVD5kiDuGXwRG65/nTkZWlLylY7s+Qw2k=
github.com/carolynvs/liquid v1.2.5-0.20220131221838-2e107bef298f h1:VQBTZqr7lKJ7I5aibSzwlDd9QiPIJ2vZCYlHB4tUfuQ=
github.com/carolynvs/liquid v1.2.5-0.20220131221838-2e107bef298f/go.mod h1:w8U5mURyI2WkBkOqadQ8C2W+oK+8TDGwo8V612sRSAI=
github.com/carolynvs/magex v0.6.0 h1:rzz4RnBiR8hr2WYEsmq+mqkRLEstPnEK8ZP9MgxNY9Y=
github.com/carolynvs/magex v0.6.0/go.mod h1:hqaEkr9TAv+kFb/5wgDiTdszF13rpe0Q+bWHmTe6N74=
github.com/casbin/casbin/v2 v2.1.2/go.mod h1:YcPU1XXisHhLzuxH9coDNf2FbKpjGlbCg3n9yuLkIJQ=
Expand Down Expand Up @@ -1184,6 +1186,8 @@ github.com/openzipkin/zipkin-go v0.1.3/go.mod h1:NtoC/o8u3JlF1lSlyPNswIbeQH9bJTm
github.com/openzipkin/zipkin-go v0.1.6/go.mod h1:QgAqvLzwWbR/WpD4A3cGpPtJrZXNIiJc5AZX7/PBEpw=
github.com/openzipkin/zipkin-go v0.2.1/go.mod h1:NaW6tEwdmWMaCDZzg8sh+IBNOxHMPnhQw8ySjnjRyN4=
github.com/openzipkin/zipkin-go v0.2.2/go.mod h1:NaW6tEwdmWMaCDZzg8sh+IBNOxHMPnhQw8ySjnjRyN4=
github.com/osteele/tuesday v1.0.3 h1:SrCmo6sWwSgnvs1bivmXLvD7Ko9+aJvvkmDjB5G4FTU=
github.com/osteele/tuesday v1.0.3/go.mod h1:pREKpE+L03UFuR+hiznj3q7j3qB1rUZ4XfKejwWFF2M=
github.com/pact-foundation/pact-go v1.0.4/go.mod h1:uExwJY4kCzNPcHRj+hCR/HBbOOIwwtUjcrb0b5/5kLM=
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k=
Expand Down
Loading