Re-organize Habitat filesystem paths, environment variables, Plan variables, and runtime templating data #374

fnichol · 2016-04-11T22:16:28Z

Background

This change set purges many more references to bldr and the current filesystem paths, namely /opt/bldr/*. The changes here aim to make a path-renaming an easy task, in as few places as possible while simultaneously creating consistent variable naming for Plan authors when they write plan.sh files and consume service config data in configuration file templates.

I have attempted to hit all all locations where these paths and variable names are impacted but I can't guarantee that it's 100% error-free. As the new template data (such as {{pkg.svc_config_path}}) is being used in the Plans, a correspondingly current Supervisor needs to be used (it produces the service config).

Here is a brief example of the pkg section of runtime config toml:

[pkg]
exposes = ["6379"]
ident = "chef/redis/3.0.7/20160405144915"
name = "redis"
origin = "chef"
path = "/opt/bldr/pkgs/chef/redis/3.0.7/20160405144915"
release = "20160405144915"
svc_config_path = "/opt/bldr/svc/redis/config"
svc_data_path = "/opt/bldr/svc/redis/data"
svc_files_path = "/opt/bldr/svc/redis/files"
svc_path = "/opt/bldr/svc/redis"
svc_static_path = "/opt/bldr/svc/redis/static"
svc_var_path = "/opt/bldr/svc/redis/var"
version = "3.0.7"

Variables

HAB_ROOT_PATH

HAB_ROOT_PATH is the The root of the Habitat tree. It currently
defaults to a value of /opt/bldr -- which will be subject to future
change. (Formerly BLDR_ROOT)

When used in shell code: $HAB_ROOT_PATH
When used in Rust code: hcore::fs::ROOT_PATH: 'static String

HAB_CACHE_ARTIFACT_PATH

HAB_CACHE_ARTIFACT_PATH is the default download root path for package artifacts, used on package installation. It defaults to $HAB_ROOT_PATH/cache/artifacts. (Formerly BLDR_PKG_CACHE)

When used in shell code: $HAB_CACHE_ARTIFACT_PATH
When used in Rust code: hcore::fs::CACHE_ARTIFACT_PATH: 'static String

HAB_CACHE_GPG_PATH

HAB_CACHE_GPG_PATH is the default path where gpg keys are stored. It
defaults to $HAB_ROOT_PATH/cache/gpg. (Formerly BLDR_GPG_CACHE)

When used in shell code: $HAB_CACHE_GPG_PATH
When used in Rust code: hcore::fs::CACHE_GPG_PATH: 'static String

HAB_CACHE_KEY_PATH

HAB_CACHE_KEY_PATH is the default path where cryptographic keys are
stored. It defaults to $HAB_ROOT_PATH/cache/keys. (Formerly HABITAT_KEY_CACHE)

When used in shell code: $HAB_CACHE_KEY_PATH
When used in Rust code: hcore::fs::CACHE_KEY_PATH: 'static String

HAB_CACHE_SRC_PATH

HAB_CACHE_SRC_PATH is the default path where source artifacts are downloaded, extracted, & compiled. It defaults to $HAB_ROOT_PATH/cache/src. (Formerly BLDR_SRC_CACHE)

When used in shell code: $HAB_CACHE_SRC_PATH
When used in Rust code: hcore::fs::CACHE_SRC_PATH: 'static String

HAB_PKG_PATH

HAB_PKG_PATH is the root path containing all locally installed packages. If defaults to $HAB_ROOT_PATH/pkgs. (Formerly BLDR_PKG_ROOT)

When used in shell code: $HAB_PKG_PATH
When used in Rust code: hcore::fs::PKG_PATH: 'static String

Service Paths

Service paths are "dynamically" generated, as they depend on the name of the package identifer.

svc_path

pkg_svc_path, {{pkg.svc_path}}, and fn svc_path() are all representations of the root path for a given service's configuration, files, and data. It is $HAB_ROOT_PATH/svc/$NAME where $NAME is the name component of a service identifier. For example, the svc_path for the chef/redis package would be $HAB_ROOT_PATH/svc/redis.

When used in Plans: $pkg_svc_path
When used in templates {{pkg.svc_path}}
When used in Rust code: PackageInstall#svc_path() -> PathBuf or hcore::fs::svc_path(&str) -> PathBuf

svc_config_path

pkg_svc_config_path, {{pkg.svc_config_path}}, and fn svc_config_path() are all representations of the path to to a given service's config. It is $pkg_svc_path/config.

When used in Plans: $pkg_svc_config_path
When used in templates: {{pkg.svc_config_path}}
When used in Rust code: PackageInstall#svc_config_path() -> PathBuf or hcore::fs::svc_config_path(&str) -> PathBuf

svc_data_path

pkg_svc_data_path, {{pkg.svc_data_path}}, and fn svc_data_path() are all representations of the path to to a given service's data. It is $pkg_svc_path/data.

When used in Plans: $pkg_svc_data_path
When used in templates: {{pkg.svc_data_path}}
When used in Rust code: PackageInstall#svc_data_path() -> PathBuf or hcore::fs::svc_data_path(&str) -> PathBuf

svc_files_path

pkg_svc_files_path, {{pkg.svc_files_path}}, and fn svc_files_path() are all representations of the path to to a given service's gossiped config files. It is $pkg_svc_path/files.

When used in Plans: $pkg_svc_files_path
When used in templates: {{pkg.svc_files_path}}
When used in Rust code: PackageInstall#svc_files_path() -> PathBuf or hcore::fs::svc_files_path(&str) -> PathBuf

svc_hooks_path

pkg_svc_hooks_path, {{pkg.svc_hooks_path}}, and fn svc_hooks_path() are all representations of the path to to a given service's hooks. It is $pkg_svc_path/hooks.

When used in Plans: Not available -- this is internal to the Supervisor
When used in templates: Not available -- this is internal to the Supervisor
When used in Rust code: PackageInstall#svc_hooks_path() -> PathBuf or hcore::fs::svc_hooks_path(&str) -> PathBuf

svc_static_path

pkg_svc_static_path, {{pkg.svc_static_path}}, and fn svc_static_path() are all representations of the path to to a given service's static content. It is $pkg_svc_path/static.

When used in Plans: $pkg_svc_static_path
When used in templates: {{pkg.svc_static_path}}
When used in Rust code: PackageInstall#svc_static_path() -> PathBuf or hcore::fs::svc_static_path(&str) -> PathBuf

svc_var_path

pkg_svc_var_path, {{pkg.svc_var_path}}, and fn svc_var_path() are all representations of the path to to a given service's variable state. It is $pkg_svc_path/var.

When used in Plans: $pkg_svc_var_path
When used in templates: {{pkg.svc_var_path}}
When used in Rust code: PackageInstall#svc_var_path() -> PathBuf or hcore::fs::svc_var_path(&str) -> PathBuf

svc_toml_path

Note I don't think this is used at the moment and so is not fully implemented.

This change seeks to remove as many hard-coded references to the `$BLDR_ROOT` of the Habitat file system tree. Several programs and utilities require this knowledge but all should now compute all other paths from this foundational one. Several occurences of the above path have been preserved as some parts of the codebase will be going away and others are too old to have been sufficiently refactored (note: these are mostly Plans that may not be supported going forward).

`HAB_ROOT_PATH` is the The root of the Habitat tree. It currently defaults to a value of `/opt/bldr` -- which will be subject to future change.

`HAB_SRC_PATH` is the default path where source artifacts are downloaded, extracted, & compiled. It defaults to `$HAB_ROOT_PATH/cache/src`.

`HAB_CACHE_ARTIFACT_PATH` is the default download root path for package artifacts, used on package installation. It defaults to `$HAB_ROOT_PATH/cache/artifacts`.

… packages. `HAB_PKG_PATH` is the root path containing all locally installed packages. If defaults to `$HAB_ROOT_PATH/pkgs`.

`HAB_CACHE_KEY_PATH` is the default path where cryptographic keys are stored. It defaults to `$HAB_ROOT_PATH/cache/keys`.

`HAB_CACHE_GPG_PATH` is the default path where gpg keys are stored. It defaults to `$HAB_ROOT_PATH/cache/gpg`.

`SVC_PATH` and `pkg_svc_path` are both representations of the root path containing all runtime service directories and files. It defaults to `$HAB_ROOT_PATH/svc`.

Also, several outstanding service config paths were found and updated.

Note that this path is internal to the Supervisor and should not be directly accessed under normal circumstances.

This change removes another source of either hardcoded paths or magic (i.e. underdocumented) directories under the service path. Instances of `PackageInstall` know their own `svc_path` as well as other flavors and this change keeps that path-building responsibility there.

chef-delivery · 2016-04-11T22:31:46Z

This PR has passed 'Verify' and is ready for review and approval!
Use: '@delivery approve' when code review is complete.

reset · 2016-04-11T22:38:07Z

@fnichol if there are any errors I willingly accept them. Thanks for doing this chore for the team, it's a big change but also a big improvement!

bookshelfdave · 2016-04-11T22:44:32Z

might be a good time to get rid of HAB_CACHE_GPG_PATH

On Apr 11, 2016, at 6:16 PM, Fletcher Nichol [email protected] wrote:

HAB_CACHE_GPG_PATH

fnichol · 2016-04-11T22:49:24Z

@metadave I was wondering if it was too early or not for that. I could add a commit, or have one ready to purge that var from the codebase

jtimberman · 2016-04-11T22:50:50Z

components/bpm/bin/hab-bpm.sh

@@ -150,10 +150,10 @@ FLAGS:
    -h  Prints this message

 OPTIONS:
-    -u <BLDR_REPO>  Sets a Bldr repository URL
+    -u <BLDR_REPO>  Sets a Habitat repository URL


Lots of BLDR_REPO, perhaps we should s/BLDR_REPO/HAB_REPO/ while we're in here.

We'll deal with this one in another PR. I wanted to make sure that "repo" is no longer the correct word.

bookshelfdave · 2016-04-12T00:25:14Z

@fnichol don't let the GPG thing hold this PR up, it's easy enough to deal with when I get the other stuff merged. Cheers!

reset · 2016-04-12T09:37:09Z

Gonna approve this one for now because I believe this is going to be a multiplass type of thing in the first place. @fnichol great work bud

@delivery approve

chef-delivery · 2016-04-12T09:37:14Z

Change: 21c0986f-7ce2-49a2-ba88-2501fc72a7f9 approved by: @reset

fnichol added 19 commits April 11, 2016 15:31

[core] Fix unintended doc unit test which needed a text escape.

a1e64d2

Use HAB_ROOT_PATH for root of Habitat file system.

1dd7758

`HAB_ROOT_PATH` is the The root of the Habitat tree. It currently defaults to a value of `/opt/bldr` -- which will be subject to future change.

Use HAB_CACHE_SRC_PATH for default src download, extract, & compile.

c35c753

`HAB_SRC_PATH` is the default path where source artifacts are downloaded, extracted, & compiled. It defaults to `$HAB_ROOT_PATH/cache/src`.

Use HAB_CACHE_ARTIFACT_PATH for download path of package artifacts.

fe1688f

`HAB_CACHE_ARTIFACT_PATH` is the default download root path for package artifacts, used on package installation. It defaults to `$HAB_ROOT_PATH/cache/artifacts`.

Use HAB_PKG_PATH for the root path containing all locally installed…

f59622f

… packages. `HAB_PKG_PATH` is the root path containing all locally installed packages. If defaults to `$HAB_ROOT_PATH/pkgs`.

Use HAB_CACHE_KEY_PATH for the default path where keys are stored.

cae7ceb

`HAB_CACHE_KEY_PATH` is the default path where cryptographic keys are stored. It defaults to `$HAB_ROOT_PATH/cache/keys`.

Use HAB_CACHE_GPG_PATH for the default path where gpg keys are stored.

f9f05ed

`HAB_CACHE_GPG_PATH` is the default path where gpg keys are stored. It defaults to `$HAB_ROOT_PATH/cache/gpg`.

Use SVC_PATH/pkg_svc_path for the root path with runtime data.

d5abf73

`SVC_PATH` and `pkg_svc_path` are both representations of the root path containing all runtime service directories and files. It defaults to `$HAB_ROOT_PATH/svc`.

SERVICE_PATH -> SVC_PATH

10c80e2

Use pkg_svc_config_path for the path to the service config.

af49ff9

Add {{pkg.svc_config_path}} to ServiceConfig.

8b60806

Use pkg_svc_data_path for the path to the service data.

c8921df

Also, several outstanding service config paths were found and updated.

Use pkg_svc_files_path for the path to the gossiped config files.

abac38a

Use pkg_svc_hooks_path for the path to the service hooks.

e73b574

Note that this path is internal to the Supervisor and should not be directly accessed under normal circumstances.

Use pkg_svc_static_path for the path to the service static content.

93eb72d

Use pkg_svc_var_path for the path to the service variable state.

168b134

Remove unneeded imports.

307d049

chef-delivery added the Delivery Change Created label Apr 11, 2016

jtimberman reviewed Apr 11, 2016
View reviewed changes

chef-delivery merged commit a2da023 into master Apr 12, 2016

chef-delivery deleted the fnichol/hab-paths-reorg branch April 12, 2016 09:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Re-organize Habitat filesystem paths, environment variables, Plan variables, and runtime templating data #374

Re-organize Habitat filesystem paths, environment variables, Plan variables, and runtime templating data #374

fnichol commented Apr 11, 2016

chef-delivery commented Apr 11, 2016

reset commented Apr 11, 2016

bookshelfdave commented Apr 11, 2016

fnichol commented Apr 11, 2016

jtimberman Apr 11, 2016

fnichol Apr 12, 2016

bookshelfdave commented Apr 12, 2016

reset commented Apr 12, 2016

chef-delivery commented Apr 12, 2016

Re-organize Habitat filesystem paths, environment variables, Plan variables, and runtime templating data #374

Re-organize Habitat filesystem paths, environment variables, Plan variables, and runtime templating data #374

Conversation

fnichol commented Apr 11, 2016

Background

Variables

HAB_ROOT_PATH

HAB_CACHE_ARTIFACT_PATH

HAB_CACHE_GPG_PATH

HAB_CACHE_KEY_PATH

HAB_CACHE_SRC_PATH

HAB_PKG_PATH

Service Paths

svc_path

svc_config_path

svc_data_path

svc_files_path

svc_hooks_path

svc_static_path

svc_var_path

svc_toml_path

chef-delivery commented Apr 11, 2016

reset commented Apr 11, 2016

bookshelfdave commented Apr 11, 2016

fnichol commented Apr 11, 2016

jtimberman Apr 11, 2016

Choose a reason for hiding this comment

fnichol Apr 12, 2016

Choose a reason for hiding this comment

bookshelfdave commented Apr 12, 2016

reset commented Apr 12, 2016

chef-delivery commented Apr 12, 2016