[hab/sup] Find suitable cacerts for HTTP/SSL client calls. #652
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change is primarily driven by a desire to front a Depot with SSL, requiring that all API calls now require a client-side set of root certificates. The challenge with the Linux `hab` CLI is that it can't necessary rely on a consistent location to find a `cacert.pem` or certificate directory. That is, it could be running on any number of 64-bit Linux operating systems, in minimal containers, etc. We are striving for a consistent user experience with as little setup as possible and ideally no extra wrangling with SSL certificates. The following is pulled from the `http-client` component's developer documentation, but applies to the behavior of `hab` and secondarily for `hab-sup` and `hab-director` at runtime. Linux Platforms Certificate Strategy ------------------------------------ We need a set of root certificates when connected to SSL/TLS web endpoints and this usually boild down to using an all-in-one certificate file (such as a `cacert.pem` file) or a directory of files which are certificates. The strategy to location or use a reasonable set of certificates is as follows: 1. If the `SSL_CERT_FILE` environment variable is set, then use its value for the certificates. Interally this is triggering default OpenSSL behavior for this environment variable. 2. If the `SSL_CERT_DIR` environment variable is set, then use its value for the directory containing certificates. Like the `SSL_CERT_FILE` case above, this triggers default OpenSSL behavior for this environment variable. 3. If the `core/cacerts` Habitat package is installed locally, then use the latest release's `cacert.pem` file. 4. If none of the conditions above are met, then a `cacert.pem` will be written in an SSL cache directory (by default `/hab/cache/ssl` for a root user and `$HOME/.hab/cache/ssl` for a non-root user) and this will be used. The contents of this file will be inlined in this crate at build time as a fallback insurance policy, meaning that if the a program using this code is operating in a minimal environment which may not contain system certificates, it can still operate. Once a `core/cacerts` Habitat package is present, the logic would fall back to preferring the package version to the cached/inline file version. Mac Platforms Certificate Strategy ---------------------------------- The Mac platoform uses a Security Framework to store and find root certificates and the hyper library will default to using this on the Mac. Therefore the behavior on the Mac remains unchanged and will use the system's certificates. Example In Minimal Linux Environment ------------------------------------ Let's use a build of `core/hab-static` in a stock Alpine Linux Docker container which by default ships with *no* SSL root certificates (these are available by running `apk --update upgrade && apk add curl ca-certificates`) First, we'll install `hab` on `PATH`, set an SSL Depot endpoint, and ensure that there is no cache (note that the `RUST_LOG` is set so we can see the resulting URLs and cacert loading logic): ``` > docker run --rm -ti -v `pwd`/results:/src alpine sh / # tail -n +6 /src/core-hab-static-*-x86_64-linux.hart | xzcat | tar x -C / / # /hab/pkgs/core/hab-static/*/*/bin/hab pkg binlink core/hab-static hab » Symlinking hab from core/hab-static into /bin ★ Binary hab from core/hab-static/0.6.0/20160604143704 symlinked to /bin/hab / # export RUST_LOG=info,habitat_http_client=debug,habitat_depot_client=debug / # export HAB_DEPOT_URL=https://willem-elb.habitat.sh/v1/depot / # ls /hab/pkgs/core hab-static / # ls /hab/cache ls: /hab/cache: No such file or directory ``` Now we'll install a package. Again, there are *no* SSL root certificates available to us (note that some DEBUG lines were removed for clarity): ``` / # hab install core/busybox-static » Installing core/busybox-static DEBUG:habitat_http_client::ssl: Creating cached cacert.pem at: /hab/cache/ssl/cert.pem DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/busybox-static/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/busybox-static/1.24.2/20160427212720 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/busybox-static/1.24.2/20160427212720/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 510.75 KB / 510.75 KB | [==============] 100.00 % 642.74 KB/s ↓ Downloading core-20160423193745 public origin key DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/origins/core/keys/20160423193745 with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 75 B / 75 B | [================] 100.00 % 1.19 MB/s ☑ Cached core-20160423193745 public origin key ✓ Installed core/busybox-static/1.24.2/20160427212720 ★ Install of core/busybox-static complete with 1 packages installed. / # ls /hab/cache/ssl/ cert.pem ``` You can see in the above output that we now have a `/hab/cache/ssl/cacert.pem` which was used to make the Depot API calls. Now we'll install the `core/cacerts` package. As before, `hab` will used the cached `cacert.pem` file: ``` / # hab install core/cacerts » Installing core/cacerts DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/cacerts/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/cacerts/2016.04.20/20160427211220 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/cacerts/2016.04.20/20160427211220/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 132.02 KB / 132.02 KB | [=======] 100.00 % 404.67 KB/s ✓ Installed core/cacerts/2016.04.20/20160427211220 ★ Install of core/cacerts complete with 1 packages installed. ``` Finally we'll install one last package, but this time `core/cacerts` is installed and available, so `hab` will use the certificates from that package instead, as they are most likely the same or newer than our inlined/cached version: ``` / # hab install core/jq-static » Installing core/jq-static DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/pkgs/core/cacerts/2016.04.20/20160427211220/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/jq-static/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/jq-static/1.5/20160427212745 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/pkgs/core/cacerts/2016.04.20/20160427211220/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/jq-static/1.5/20160427212745/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 828.97 KB / 828.97 KB / [=======] 100.00 % 392.74 KB/s ✓ Installed core/jq-static/1.5/20160427212745 ★ Install of core/jq-static complete with 1 packages installed. ``` Signed-off-by: Fletcher Nichol <[email protected]>
This change is admittedly a little sad, but is pragmatic in the short term. The current releases of the Hyper crate (which is the underlying HTTP client implementation) doesn't provide an API to construct a client with a particular SSL context *and* http_proxy support. As a secured Depot communication will trump most other features, we decided to disable http_proxy support until the team can get a change upstream to support this desired setup. The previous commit (and therefore this one) will serve as the rough implementation of `new_hyper_client` when we can restore proxy support. Signed-off-by: Fletcher Nichol <[email protected]>
|
By analyzing the blame information on this pull request, we identified @reset, @adamhjk, @metadave, @smith and @jtimberman to be potential reviewers |
|
This should also be known as the "Have Certs, Will Travel" pull request
|
|
For reviewers note that the |
|
|
|
📌 Commit 7351249 has been approved by |
thesentinels
pushed a commit
that referenced
this pull request
Jun 4, 2016
This change is primarily driven by a desire to front a Depot with SSL, requiring that all API calls now require a client-side set of root certificates. The challenge with the Linux `hab` CLI is that it can't necessary rely on a consistent location to find a `cacert.pem` or certificate directory. That is, it could be running on any number of 64-bit Linux operating systems, in minimal containers, etc. We are striving for a consistent user experience with as little setup as possible and ideally no extra wrangling with SSL certificates. The following is pulled from the `http-client` component's developer documentation, but applies to the behavior of `hab` and secondarily for `hab-sup` and `hab-director` at runtime. Linux Platforms Certificate Strategy ------------------------------------ We need a set of root certificates when connected to SSL/TLS web endpoints and this usually boild down to using an all-in-one certificate file (such as a `cacert.pem` file) or a directory of files which are certificates. The strategy to location or use a reasonable set of certificates is as follows: 1. If the `SSL_CERT_FILE` environment variable is set, then use its value for the certificates. Interally this is triggering default OpenSSL behavior for this environment variable. 2. If the `SSL_CERT_DIR` environment variable is set, then use its value for the directory containing certificates. Like the `SSL_CERT_FILE` case above, this triggers default OpenSSL behavior for this environment variable. 3. If the `core/cacerts` Habitat package is installed locally, then use the latest release's `cacert.pem` file. 4. If none of the conditions above are met, then a `cacert.pem` will be written in an SSL cache directory (by default `/hab/cache/ssl` for a root user and `$HOME/.hab/cache/ssl` for a non-root user) and this will be used. The contents of this file will be inlined in this crate at build time as a fallback insurance policy, meaning that if the a program using this code is operating in a minimal environment which may not contain system certificates, it can still operate. Once a `core/cacerts` Habitat package is present, the logic would fall back to preferring the package version to the cached/inline file version. Mac Platforms Certificate Strategy ---------------------------------- The Mac platoform uses a Security Framework to store and find root certificates and the hyper library will default to using this on the Mac. Therefore the behavior on the Mac remains unchanged and will use the system's certificates. Example In Minimal Linux Environment ------------------------------------ Let's use a build of `core/hab-static` in a stock Alpine Linux Docker container which by default ships with *no* SSL root certificates (these are available by running `apk --update upgrade && apk add curl ca-certificates`) First, we'll install `hab` on `PATH`, set an SSL Depot endpoint, and ensure that there is no cache (note that the `RUST_LOG` is set so we can see the resulting URLs and cacert loading logic): ``` > docker run --rm -ti -v `pwd`/results:/src alpine sh / # tail -n +6 /src/core-hab-static-*-x86_64-linux.hart | xzcat | tar x -C / / # /hab/pkgs/core/hab-static/*/*/bin/hab pkg binlink core/hab-static hab » Symlinking hab from core/hab-static into /bin ★ Binary hab from core/hab-static/0.6.0/20160604143704 symlinked to /bin/hab / # export RUST_LOG=info,habitat_http_client=debug,habitat_depot_client=debug / # export HAB_DEPOT_URL=https://willem-elb.habitat.sh/v1/depot / # ls /hab/pkgs/core hab-static / # ls /hab/cache ls: /hab/cache: No such file or directory ``` Now we'll install a package. Again, there are *no* SSL root certificates available to us (note that some DEBUG lines were removed for clarity): ``` / # hab install core/busybox-static » Installing core/busybox-static DEBUG:habitat_http_client::ssl: Creating cached cacert.pem at: /hab/cache/ssl/cert.pem DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/busybox-static/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/busybox-static/1.24.2/20160427212720 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/busybox-static/1.24.2/20160427212720/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 510.75 KB / 510.75 KB | [==============] 100.00 % 642.74 KB/s ↓ Downloading core-20160423193745 public origin key DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/origins/core/keys/20160423193745 with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 75 B / 75 B | [================] 100.00 % 1.19 MB/s ☑ Cached core-20160423193745 public origin key ✓ Installed core/busybox-static/1.24.2/20160427212720 ★ Install of core/busybox-static complete with 1 packages installed. / # ls /hab/cache/ssl/ cert.pem ``` You can see in the above output that we now have a `/hab/cache/ssl/cacert.pem` which was used to make the Depot API calls. Now we'll install the `core/cacerts` package. As before, `hab` will used the cached `cacert.pem` file: ``` / # hab install core/cacerts » Installing core/cacerts DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/cacerts/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/cacerts/2016.04.20/20160427211220 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/cacerts/2016.04.20/20160427211220/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 132.02 KB / 132.02 KB | [=======] 100.00 % 404.67 KB/s ✓ Installed core/cacerts/2016.04.20/20160427211220 ★ Install of core/cacerts complete with 1 packages installed. ``` Finally we'll install one last package, but this time `core/cacerts` is installed and available, so `hab` will use the certificates from that package instead, as they are most likely the same or newer than our inlined/cached version: ``` / # hab install core/jq-static » Installing core/jq-static DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/pkgs/core/cacerts/2016.04.20/20160427211220/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/jq-static/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/jq-static/1.5/20160427212745 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/pkgs/core/cacerts/2016.04.20/20160427211220/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/jq-static/1.5/20160427212745/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 828.97 KB / 828.97 KB / [=======] 100.00 % 392.74 KB/s ✓ Installed core/jq-static/1.5/20160427212745 ★ Install of core/jq-static complete with 1 packages installed. ``` Signed-off-by: Fletcher Nichol <[email protected]> Pull request: #652 Approved by: adamhjk
thesentinels
pushed a commit
that referenced
this pull request
Jun 4, 2016
This change is admittedly a little sad, but is pragmatic in the short term. The current releases of the Hyper crate (which is the underlying HTTP client implementation) doesn't provide an API to construct a client with a particular SSL context *and* http_proxy support. As a secured Depot communication will trump most other features, we decided to disable http_proxy support until the team can get a change upstream to support this desired setup. The previous commit (and therefore this one) will serve as the rough implementation of `new_hyper_client` when we can restore proxy support. Signed-off-by: Fletcher Nichol <[email protected]> Pull request: #652 Approved by: adamhjk
|
☀️ Test successful - travis |
jtimberman
pushed a commit
that referenced
this pull request
Jun 12, 2016
This change is primarily driven by a desire to front a Depot with SSL, requiring that all API calls now require a client-side set of root certificates. The challenge with the Linux `hab` CLI is that it can't necessary rely on a consistent location to find a `cacert.pem` or certificate directory. That is, it could be running on any number of 64-bit Linux operating systems, in minimal containers, etc. We are striving for a consistent user experience with as little setup as possible and ideally no extra wrangling with SSL certificates. The following is pulled from the `http-client` component's developer documentation, but applies to the behavior of `hab` and secondarily for `hab-sup` and `hab-director` at runtime. Linux Platforms Certificate Strategy ------------------------------------ We need a set of root certificates when connected to SSL/TLS web endpoints and this usually boild down to using an all-in-one certificate file (such as a `cacert.pem` file) or a directory of files which are certificates. The strategy to location or use a reasonable set of certificates is as follows: 1. If the `SSL_CERT_FILE` environment variable is set, then use its value for the certificates. Interally this is triggering default OpenSSL behavior for this environment variable. 2. If the `SSL_CERT_DIR` environment variable is set, then use its value for the directory containing certificates. Like the `SSL_CERT_FILE` case above, this triggers default OpenSSL behavior for this environment variable. 3. If the `core/cacerts` Habitat package is installed locally, then use the latest release's `cacert.pem` file. 4. If none of the conditions above are met, then a `cacert.pem` will be written in an SSL cache directory (by default `/hab/cache/ssl` for a root user and `$HOME/.hab/cache/ssl` for a non-root user) and this will be used. The contents of this file will be inlined in this crate at build time as a fallback insurance policy, meaning that if the a program using this code is operating in a minimal environment which may not contain system certificates, it can still operate. Once a `core/cacerts` Habitat package is present, the logic would fall back to preferring the package version to the cached/inline file version. Mac Platforms Certificate Strategy ---------------------------------- The Mac platoform uses a Security Framework to store and find root certificates and the hyper library will default to using this on the Mac. Therefore the behavior on the Mac remains unchanged and will use the system's certificates. Example In Minimal Linux Environment ------------------------------------ Let's use a build of `core/hab-static` in a stock Alpine Linux Docker container which by default ships with *no* SSL root certificates (these are available by running `apk --update upgrade && apk add curl ca-certificates`) First, we'll install `hab` on `PATH`, set an SSL Depot endpoint, and ensure that there is no cache (note that the `RUST_LOG` is set so we can see the resulting URLs and cacert loading logic): ``` > docker run --rm -ti -v `pwd`/results:/src alpine sh / # tail -n +6 /src/core-hab-static-*-x86_64-linux.hart | xzcat | tar x -C / / # /hab/pkgs/core/hab-static/*/*/bin/hab pkg binlink core/hab-static hab » Symlinking hab from core/hab-static into /bin ★ Binary hab from core/hab-static/0.6.0/20160604143704 symlinked to /bin/hab / # export RUST_LOG=info,habitat_http_client=debug,habitat_depot_client=debug / # export HAB_DEPOT_URL=https://willem-elb.habitat.sh/v1/depot / # ls /hab/pkgs/core hab-static / # ls /hab/cache ls: /hab/cache: No such file or directory ``` Now we'll install a package. Again, there are *no* SSL root certificates available to us (note that some DEBUG lines were removed for clarity): ``` / # hab install core/busybox-static » Installing core/busybox-static DEBUG:habitat_http_client::ssl: Creating cached cacert.pem at: /hab/cache/ssl/cert.pem DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/busybox-static/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/busybox-static/1.24.2/20160427212720 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/busybox-static/1.24.2/20160427212720/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 510.75 KB / 510.75 KB | [==============] 100.00 % 642.74 KB/s ↓ Downloading core-20160423193745 public origin key DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/origins/core/keys/20160423193745 with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 75 B / 75 B | [================] 100.00 % 1.19 MB/s ☑ Cached core-20160423193745 public origin key ✓ Installed core/busybox-static/1.24.2/20160427212720 ★ Install of core/busybox-static complete with 1 packages installed. / # ls /hab/cache/ssl/ cert.pem ``` You can see in the above output that we now have a `/hab/cache/ssl/cacert.pem` which was used to make the Depot API calls. Now we'll install the `core/cacerts` package. As before, `hab` will used the cached `cacert.pem` file: ``` / # hab install core/cacerts » Installing core/cacerts DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/cacerts/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/cacerts/2016.04.20/20160427211220 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/cacerts/2016.04.20/20160427211220/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 132.02 KB / 132.02 KB | [=======] 100.00 % 404.67 KB/s ✓ Installed core/cacerts/2016.04.20/20160427211220 ★ Install of core/cacerts complete with 1 packages installed. ``` Finally we'll install one last package, but this time `core/cacerts` is installed and available, so `hab` will use the certificates from that package instead, as they are most likely the same or newer than our inlined/cached version: ``` / # hab install core/jq-static » Installing core/jq-static DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/pkgs/core/cacerts/2016.04.20/20160427211220/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/jq-static/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/jq-static/1.5/20160427212745 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/pkgs/core/cacerts/2016.04.20/20160427211220/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/jq-static/1.5/20160427212745/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 828.97 KB / 828.97 KB / [=======] 100.00 % 392.74 KB/s ✓ Installed core/jq-static/1.5/20160427212745 ★ Install of core/jq-static complete with 1 packages installed. ``` Signed-off-by: Fletcher Nichol <[email protected]> Pull request: #652 Approved by: adamhjk
jtimberman
pushed a commit
that referenced
this pull request
Jun 12, 2016
This change is admittedly a little sad, but is pragmatic in the short term. The current releases of the Hyper crate (which is the underlying HTTP client implementation) doesn't provide an API to construct a client with a particular SSL context *and* http_proxy support. As a secured Depot communication will trump most other features, we decided to disable http_proxy support until the team can get a change upstream to support this desired setup. The previous commit (and therefore this one) will serve as the rough implementation of `new_hyper_client` when we can restore proxy support. Signed-off-by: Fletcher Nichol <[email protected]> Pull request: #652 Approved by: adamhjk
fnichol
added a commit
that referenced
this pull request
Jul 6, 2016
This change re-introduces support for using HTTP proxies when contacting a Depot service. Previous versions of the hyper Rust crate were not sufficiently flexible to support both custom SSL contexts and proxy support which led to the removal of proxy support in #652. This new strategy adds support back by using a custom Hyper `NetworkConnector` that supports plaintext (i.e. http) and tunneled (i.e. https) proxying modes as well as optional basic authorization support. All 4 variants should cover the vast majority of corporate network/proxy setups. Background Reading Materials ---------------------------- Several references, documents, and RFCs were used in working up this feature and are shared here for context and for the curious: * RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1" https://tools.ietf.org/html/rfc2616 * RFC 2617 - "HTTP Authentication: Basic and Digest Access Authentication" https://tools.ietf.org/html/rfc2617 * RFC 2817 - "Upgrading to TLS Within HTTP/1.1" https://tools.ietf.org/html/rfc2817 * RFC draft - "Tunneling TCP based protocols through Web proxy servers" http://www.web-cache.com/Writings/Internet-Drafts/draft-luotonen-web-proxy-tunneling-01.txt * Polipo Manual - "Access Control" https://www.irif.univ-paris-diderot.fr/~jch/software/polipo/polipo.html#index-authCredentials * Squid Cache Wiki - "Feature: HTTPS (HTTP Secure or HTTP over SSL/TLS)" http://wiki.squid-cache.org/Features/HTTPS * curl Man Page - "Environment" https://curl.haxx.se/docs/manpage.html * Wget Manual - "Proxies" https://www.gnu.org/software/wget/manual/html_node/Proxies.html * Arch Linux Wiki - "Proxy settings" https://wiki.archlinux.org/index.php/proxy_settings Environment Variable Behavior ----------------------------- * `http_proxy` / `HTTP_PROXY` - A URL for a proxy server that will be used for any HTTP connections. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment variable takes precedence over the uppercase version. * `https_proxy` / `HTTPS_PROXY` - A URL for a proxy server that will be used for any HTTPS connections. This uses a TCP tunneling mode, meaning that the proxy server tunnels the encrypted socket connection through to the target and has no ability to intercept or unpack the contents of the connection. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment varible takes precedence over the uppercase version. * `no_proxy` / `NO_PROXY` - A comma-separated list of domain extensions that a proxy should *not* be used for. For example, if the value of `no_proxy` is `".example.com"` the a proxy should not be used for URLs ending with the `example.com` domain. The lowercase version of this environment variable takes precedence over the uppercase version. Bonus: ApiClient ---------------- In additional to supporting the above feature a refactoring of the habitat_http_client component led to the creation of an "API Client" struct to wrap up the underyling hyper HTTP behavior. It centers around a common base URL so that a proxy server can be determined and set up correctly for the target. As proxy authentication is implemented quite differently between plaintext and tunneled mode, it was easier to encapsulate this behavior inside the `ApiClient`. See the developer documentation for more usage details. Bonus: User-Agent Headers ------------------------- Finally, the wrapping of HTTP client calls behind the above mentioned `ApiClient` allows us to include a common `User-Agent` HTTP header which can be used to measure the various versions of software that connect to a Depot or other endpoints. The format of the `User-Agent` header value is similar to the analytics code implementation in the `hab` CLI. Final Word About depot-client API --------------------------------- Supporting an injection of a product name and version into each Depot client has led to the addition of function parameters in the codebase where a client is created. This isn't yet totally ideal, but future work is slated to reduce the number of Depot client creations and to simplfy its setup API. The author of this current feature asks for forgiveness in the meantime and not permission ;) Signed-off-by: Fletcher Nichol <[email protected]>
fnichol
added a commit
that referenced
this pull request
Jul 6, 2016
This change re-introduces support for using HTTP proxies when contacting a Depot service. Previous versions of the hyper Rust crate were not sufficiently flexible to support both custom SSL contexts and proxy support which led to the removal of proxy support in #652. This new strategy adds support back by using a custom Hyper `NetworkConnector` that supports plaintext (i.e. http) and tunneled (i.e. https) proxying modes as well as optional basic authorization support. All 4 variants should cover the vast majority of corporate network/proxy setups. Background Reading Materials ---------------------------- Several references, documents, and RFCs were used in working up this feature and are shared here for context and for the curious: * RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1" https://tools.ietf.org/html/rfc2616 * RFC 2617 - "HTTP Authentication: Basic and Digest Access Authentication" https://tools.ietf.org/html/rfc2617 * RFC 2817 - "Upgrading to TLS Within HTTP/1.1" https://tools.ietf.org/html/rfc2817 * RFC draft - "Tunneling TCP based protocols through Web proxy servers" http://www.web-cache.com/Writings/Internet-Drafts/draft-luotonen-web-proxy-tunneling-01.txt * Polipo Manual - "Access Control" https://www.irif.univ-paris-diderot.fr/~jch/software/polipo/polipo.html#index-authCredentials * Squid Cache Wiki - "Feature: HTTPS (HTTP Secure or HTTP over SSL/TLS)" http://wiki.squid-cache.org/Features/HTTPS * curl Man Page - "Environment" https://curl.haxx.se/docs/manpage.html * Wget Manual - "Proxies" https://www.gnu.org/software/wget/manual/html_node/Proxies.html * Arch Linux Wiki - "Proxy settings" https://wiki.archlinux.org/index.php/proxy_settings Environment Variable Behavior ----------------------------- * `http_proxy` / `HTTP_PROXY` - A URL for a proxy server that will be used for any HTTP connections. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment variable takes precedence over the uppercase version. * `https_proxy` / `HTTPS_PROXY` - A URL for a proxy server that will be used for any HTTPS connections. This uses a TCP tunneling mode, meaning that the proxy server tunnels the encrypted socket connection through to the target and has no ability to intercept or unpack the contents of the connection. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment varible takes precedence over the uppercase version. * `no_proxy` / `NO_PROXY` - A comma-separated list of domain extensions that a proxy should *not* be used for. For example, if the value of `no_proxy` is `".example.com"` the a proxy should not be used for URLs ending with the `example.com` domain. The lowercase version of this environment variable takes precedence over the uppercase version. Bonus: ApiClient ---------------- In additional to supporting the above feature a refactoring of the habitat_http_client component led to the creation of an "API Client" struct to wrap up the underyling hyper HTTP behavior. It centers around a common base URL so that a proxy server can be determined and set up correctly for the target. As proxy authentication is implemented quite differently between plaintext and tunneled mode, it was easier to encapsulate this behavior inside the `ApiClient`. See the developer documentation for more usage details. Bonus: User-Agent Headers ------------------------- Finally, the wrapping of HTTP client calls behind the above mentioned `ApiClient` allows us to include a common `User-Agent` HTTP header which can be used to measure the various versions of software that connect to a Depot or other endpoints. The format of the `User-Agent` header value is similar to the analytics code implementation in the `hab` CLI. Final Word About depot-client API --------------------------------- Supporting an injection of a product name and version into each Depot client has led to the addition of function parameters in the codebase where a client is created. This isn't yet totally ideal, but future work is slated to reduce the number of Depot client creations and to simplfy its setup API. The author of this current feature asks for forgiveness in the meantime and not permission ;) Signed-off-by: Fletcher Nichol <[email protected]>
thesentinels
added a commit
that referenced
this pull request
Jul 7, 2016
…=fnichol [hab,hab-sup,hab-director] Add http & https proxy support. This change re-introduces support for using HTTP proxies when contacting a Depot service. Previous versions of the hyper Rust crate were not sufficiently flexible to support both custom SSL contexts and proxy support which led to the removal of proxy support in #652. This new strategy adds support back by using a custom Hyper `NetworkConnector` that supports plaintext (i.e. http) and tunneled (i.e. https) proxying modes as well as optional basic authorization support. All 4 variants should cover the vast majority of corporate network/proxy setups. Background Reading Materials ---------------------------- Several references, documents, and RFCs were used in working up this feature and are shared here for context and for the curious: * RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1" https://tools.ietf.org/html/rfc2616 * RFC 2617 - "HTTP Authentication: Basic and Digest Access Authentication" https://tools.ietf.org/html/rfc2617 * RFC 2817 - "Upgrading to TLS Within HTTP/1.1" https://tools.ietf.org/html/rfc2817 * RFC draft - "Tunneling TCP based protocols through Web proxy servers" http://www.web-cache.com/Writings/Internet-Drafts/draft-luotonen-web-proxy-tunneling-01.txt * Polipo Manual - "Access Control" https://www.irif.univ-paris-diderot.fr/~jch/software/polipo/polipo.html#index-authCredentials * Squid Cache Wiki - "Feature: HTTPS (HTTP Secure or HTTP over SSL/TLS)" http://wiki.squid-cache.org/Features/HTTPS * curl Man Page - "Environment" https://curl.haxx.se/docs/manpage.html * Wget Manual - "Proxies" https://www.gnu.org/software/wget/manual/html_node/Proxies.html * Arch Linux Wiki - "Proxy settings" https://wiki.archlinux.org/index.php/proxy_settings Environment Variable Behavior ----------------------------- * `http_proxy` / `HTTP_PROXY` - A URL for a proxy server that will be used for any HTTP connections. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment variable takes precedence over the uppercase version. * `https_proxy` / `HTTPS_PROXY` - A URL for a proxy server that will be used for any HTTPS connections. This uses a TCP tunneling mode, meaning that the proxy server tunnels the encrypted socket connection through to the target and has no ability to intercept or unpack the contents of the connection. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment varible takes precedence over the uppercase version. * `no_proxy` / `NO_PROXY` - A comma-separated list of domain extensions that a proxy should *not* be used for. For example, if the value of `no_proxy` is `".example.com"` the a proxy should not be used for URLs ending with the `example.com` domain. The lowercase version of this environment variable takes precedence over the uppercase version. Bonus: ApiClient ---------------- In additional to supporting the above feature a refactoring of the habitat_http_client component led to the creation of an "API Client" struct to wrap up the underyling hyper HTTP behavior. It centers around a common base URL so that a proxy server can be determined and set up correctly for the target. As proxy authentication is implemented quite differently between plaintext and tunneled mode, it was easier to encapsulate this behavior inside the `ApiClient`. See the developer documentation for more usage details. Bonus: User-Agent Headers ------------------------- Finally, the wrapping of HTTP client calls behind the above mentioned `ApiClient` allows us to include a common `User-Agent` HTTP header which can be used to measure the various versions of software that connect to a Depot or other endpoints. The format of the `User-Agent` header value is similar to the analytics code implementation in the `hab` CLI. Final Word About depot-client API --------------------------------- Supporting an injection of a product name and version into each Depot client has led to the addition of function parameters in the codebase where a client is created. This isn't yet totally ideal, but future work is slated to reduce the number of Depot client creations and to simplfy its setup API. The author of this current feature asks for forgiveness in the meantime and not permission ;) Signed-off-by: Fletcher Nichol <[email protected]>
raskchanky
pushed a commit
that referenced
this pull request
Apr 16, 2019
This change is primarily driven by a desire to front a Depot with SSL, requiring that all API calls now require a client-side set of root certificates. The challenge with the Linux `hab` CLI is that it can't necessary rely on a consistent location to find a `cacert.pem` or certificate directory. That is, it could be running on any number of 64-bit Linux operating systems, in minimal containers, etc. We are striving for a consistent user experience with as little setup as possible and ideally no extra wrangling with SSL certificates. The following is pulled from the `http-client` component's developer documentation, but applies to the behavior of `hab` and secondarily for `hab-sup` and `hab-director` at runtime. Linux Platforms Certificate Strategy ------------------------------------ We need a set of root certificates when connected to SSL/TLS web endpoints and this usually boild down to using an all-in-one certificate file (such as a `cacert.pem` file) or a directory of files which are certificates. The strategy to location or use a reasonable set of certificates is as follows: 1. If the `SSL_CERT_FILE` environment variable is set, then use its value for the certificates. Interally this is triggering default OpenSSL behavior for this environment variable. 2. If the `SSL_CERT_DIR` environment variable is set, then use its value for the directory containing certificates. Like the `SSL_CERT_FILE` case above, this triggers default OpenSSL behavior for this environment variable. 3. If the `core/cacerts` Habitat package is installed locally, then use the latest release's `cacert.pem` file. 4. If none of the conditions above are met, then a `cacert.pem` will be written in an SSL cache directory (by default `/hab/cache/ssl` for a root user and `$HOME/.hab/cache/ssl` for a non-root user) and this will be used. The contents of this file will be inlined in this crate at build time as a fallback insurance policy, meaning that if the a program using this code is operating in a minimal environment which may not contain system certificates, it can still operate. Once a `core/cacerts` Habitat package is present, the logic would fall back to preferring the package version to the cached/inline file version. Mac Platforms Certificate Strategy ---------------------------------- The Mac platoform uses a Security Framework to store and find root certificates and the hyper library will default to using this on the Mac. Therefore the behavior on the Mac remains unchanged and will use the system's certificates. Example In Minimal Linux Environment ------------------------------------ Let's use a build of `core/hab-static` in a stock Alpine Linux Docker container which by default ships with *no* SSL root certificates (these are available by running `apk --update upgrade && apk add curl ca-certificates`) First, we'll install `hab` on `PATH`, set an SSL Depot endpoint, and ensure that there is no cache (note that the `RUST_LOG` is set so we can see the resulting URLs and cacert loading logic): ``` > docker run --rm -ti -v `pwd`/results:/src alpine sh / # tail -n +6 /src/core-hab-static-*-x86_64-linux.hart | xzcat | tar x -C / / # /hab/pkgs/core/hab-static/*/*/bin/hab pkg binlink core/hab-static hab » Symlinking hab from core/hab-static into /bin ★ Binary hab from core/hab-static/0.6.0/20160604143704 symlinked to /bin/hab / # export RUST_LOG=info,habitat_http_client=debug,habitat_depot_client=debug / # export HAB_DEPOT_URL=https://willem-elb.habitat.sh/v1/depot / # ls /hab/pkgs/core hab-static / # ls /hab/cache ls: /hab/cache: No such file or directory ``` Now we'll install a package. Again, there are *no* SSL root certificates available to us (note that some DEBUG lines were removed for clarity): ``` / # hab install core/busybox-static » Installing core/busybox-static DEBUG:habitat_http_client::ssl: Creating cached cacert.pem at: /hab/cache/ssl/cert.pem DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/busybox-static/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/busybox-static/1.24.2/20160427212720 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/busybox-static/1.24.2/20160427212720/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 510.75 KB / 510.75 KB | [==============] 100.00 % 642.74 KB/s ↓ Downloading core-20160423193745 public origin key DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/origins/core/keys/20160423193745 with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 75 B / 75 B | [================] 100.00 % 1.19 MB/s ☑ Cached core-20160423193745 public origin key ✓ Installed core/busybox-static/1.24.2/20160427212720 ★ Install of core/busybox-static complete with 1 packages installed. / # ls /hab/cache/ssl/ cert.pem ``` You can see in the above output that we now have a `/hab/cache/ssl/cacert.pem` which was used to make the Depot API calls. Now we'll install the `core/cacerts` package. As before, `hab` will used the cached `cacert.pem` file: ``` / # hab install core/cacerts » Installing core/cacerts DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/cacerts/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/cacerts/2016.04.20/20160427211220 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/cacerts/2016.04.20/20160427211220/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 132.02 KB / 132.02 KB | [=======] 100.00 % 404.67 KB/s ✓ Installed core/cacerts/2016.04.20/20160427211220 ★ Install of core/cacerts complete with 1 packages installed. ``` Finally we'll install one last package, but this time `core/cacerts` is installed and available, so `hab` will use the certificates from that package instead, as they are most likely the same or newer than our inlined/cached version: ``` / # hab install core/jq-static » Installing core/jq-static DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/pkgs/core/cacerts/2016.04.20/20160427211220/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/jq-static/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/jq-static/1.5/20160427212745 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/pkgs/core/cacerts/2016.04.20/20160427211220/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/jq-static/1.5/20160427212745/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 828.97 KB / 828.97 KB / [=======] 100.00 % 392.74 KB/s ✓ Installed core/jq-static/1.5/20160427212745 ★ Install of core/jq-static complete with 1 packages installed. ``` Signed-off-by: Fletcher Nichol <[email protected]> Pull request: #652 Approved by: adamhjk
raskchanky
pushed a commit
that referenced
this pull request
Apr 16, 2019
This change is admittedly a little sad, but is pragmatic in the short term. The current releases of the Hyper crate (which is the underlying HTTP client implementation) doesn't provide an API to construct a client with a particular SSL context *and* http_proxy support. As a secured Depot communication will trump most other features, we decided to disable http_proxy support until the team can get a change upstream to support this desired setup. The previous commit (and therefore this one) will serve as the rough implementation of `new_hyper_client` when we can restore proxy support. Signed-off-by: Fletcher Nichol <[email protected]> Pull request: #652 Approved by: adamhjk
raskchanky
pushed a commit
that referenced
this pull request
Apr 16, 2019
This change is primarily driven by a desire to front a Depot with SSL, requiring that all API calls now require a client-side set of root certificates. The challenge with the Linux `hab` CLI is that it can't necessary rely on a consistent location to find a `cacert.pem` or certificate directory. That is, it could be running on any number of 64-bit Linux operating systems, in minimal containers, etc. We are striving for a consistent user experience with as little setup as possible and ideally no extra wrangling with SSL certificates. The following is pulled from the `http-client` component's developer documentation, but applies to the behavior of `hab` and secondarily for `hab-sup` and `hab-director` at runtime. Linux Platforms Certificate Strategy ------------------------------------ We need a set of root certificates when connected to SSL/TLS web endpoints and this usually boild down to using an all-in-one certificate file (such as a `cacert.pem` file) or a directory of files which are certificates. The strategy to location or use a reasonable set of certificates is as follows: 1. If the `SSL_CERT_FILE` environment variable is set, then use its value for the certificates. Interally this is triggering default OpenSSL behavior for this environment variable. 2. If the `SSL_CERT_DIR` environment variable is set, then use its value for the directory containing certificates. Like the `SSL_CERT_FILE` case above, this triggers default OpenSSL behavior for this environment variable. 3. If the `core/cacerts` Habitat package is installed locally, then use the latest release's `cacert.pem` file. 4. If none of the conditions above are met, then a `cacert.pem` will be written in an SSL cache directory (by default `/hab/cache/ssl` for a root user and `$HOME/.hab/cache/ssl` for a non-root user) and this will be used. The contents of this file will be inlined in this crate at build time as a fallback insurance policy, meaning that if the a program using this code is operating in a minimal environment which may not contain system certificates, it can still operate. Once a `core/cacerts` Habitat package is present, the logic would fall back to preferring the package version to the cached/inline file version. Mac Platforms Certificate Strategy ---------------------------------- The Mac platoform uses a Security Framework to store and find root certificates and the hyper library will default to using this on the Mac. Therefore the behavior on the Mac remains unchanged and will use the system's certificates. Example In Minimal Linux Environment ------------------------------------ Let's use a build of `core/hab-static` in a stock Alpine Linux Docker container which by default ships with *no* SSL root certificates (these are available by running `apk --update upgrade && apk add curl ca-certificates`) First, we'll install `hab` on `PATH`, set an SSL Depot endpoint, and ensure that there is no cache (note that the `RUST_LOG` is set so we can see the resulting URLs and cacert loading logic): ``` > docker run --rm -ti -v `pwd`/results:/src alpine sh / # tail -n +6 /src/core-hab-static-*-x86_64-linux.hart | xzcat | tar x -C / / # /hab/pkgs/core/hab-static/*/*/bin/hab pkg binlink core/hab-static hab » Symlinking hab from core/hab-static into /bin ★ Binary hab from core/hab-static/0.6.0/20160604143704 symlinked to /bin/hab / # export RUST_LOG=info,habitat_http_client=debug,habitat_depot_client=debug / # export HAB_DEPOT_URL=https://willem-elb.habitat.sh/v1/depot / # ls /hab/pkgs/core hab-static / # ls /hab/cache ls: /hab/cache: No such file or directory ``` Now we'll install a package. Again, there are *no* SSL root certificates available to us (note that some DEBUG lines were removed for clarity): ``` / # hab install core/busybox-static » Installing core/busybox-static DEBUG:habitat_http_client::ssl: Creating cached cacert.pem at: /hab/cache/ssl/cert.pem DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/busybox-static/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/busybox-static/1.24.2/20160427212720 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/busybox-static/1.24.2/20160427212720/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 510.75 KB / 510.75 KB | [==============] 100.00 % 642.74 KB/s ↓ Downloading core-20160423193745 public origin key DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/origins/core/keys/20160423193745 with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 75 B / 75 B | [================] 100.00 % 1.19 MB/s ☑ Cached core-20160423193745 public origin key ✓ Installed core/busybox-static/1.24.2/20160427212720 ★ Install of core/busybox-static complete with 1 packages installed. / # ls /hab/cache/ssl/ cert.pem ``` You can see in the above output that we now have a `/hab/cache/ssl/cacert.pem` which was used to make the Depot API calls. Now we'll install the `core/cacerts` package. As before, `hab` will used the cached `cacert.pem` file: ``` / # hab install core/cacerts » Installing core/cacerts DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/cacerts/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/cacerts/2016.04.20/20160427211220 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/cacerts/2016.04.20/20160427211220/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 132.02 KB / 132.02 KB | [=======] 100.00 % 404.67 KB/s ✓ Installed core/cacerts/2016.04.20/20160427211220 ★ Install of core/cacerts complete with 1 packages installed. ``` Finally we'll install one last package, but this time `core/cacerts` is installed and available, so `hab` will use the certificates from that package instead, as they are most likely the same or newer than our inlined/cached version: ``` / # hab install core/jq-static » Installing core/jq-static DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/pkgs/core/cacerts/2016.04.20/20160427211220/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/jq-static/latest with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } ↓ Downloading core/jq-static/1.5/20160427212745 DEBUG:habitat_http_client::ssl: Setting CA file for SSL context to: /hab/pkgs/core/cacerts/2016.04.20/20160427211220/ssl/cert.pem DEBUG:habitat_depot_client: GET https://willem-elb.habitat.sh/v1/depot/pkgs/core/jq-static/1.5/20160427212745/download with Client { redirect_policy: FollowAll, read_timeout: None, write_timeout: None, proxy: None } 828.97 KB / 828.97 KB / [=======] 100.00 % 392.74 KB/s ✓ Installed core/jq-static/1.5/20160427212745 ★ Install of core/jq-static complete with 1 packages installed. ``` Signed-off-by: Fletcher Nichol <[email protected]> Pull request: #652 Approved by: adamhjk
raskchanky
pushed a commit
that referenced
this pull request
Apr 16, 2019
This change re-introduces support for using HTTP proxies when contacting a Depot service. Previous versions of the hyper Rust crate were not sufficiently flexible to support both custom SSL contexts and proxy support which led to the removal of proxy support in #652. This new strategy adds support back by using a custom Hyper `NetworkConnector` that supports plaintext (i.e. http) and tunneled (i.e. https) proxying modes as well as optional basic authorization support. All 4 variants should cover the vast majority of corporate network/proxy setups. Background Reading Materials ---------------------------- Several references, documents, and RFCs were used in working up this feature and are shared here for context and for the curious: * RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1" https://tools.ietf.org/html/rfc2616 * RFC 2617 - "HTTP Authentication: Basic and Digest Access Authentication" https://tools.ietf.org/html/rfc2617 * RFC 2817 - "Upgrading to TLS Within HTTP/1.1" https://tools.ietf.org/html/rfc2817 * RFC draft - "Tunneling TCP based protocols through Web proxy servers" http://www.web-cache.com/Writings/Internet-Drafts/draft-luotonen-web-proxy-tunneling-01.txt * Polipo Manual - "Access Control" https://www.irif.univ-paris-diderot.fr/~jch/software/polipo/polipo.html#index-authCredentials * Squid Cache Wiki - "Feature: HTTPS (HTTP Secure or HTTP over SSL/TLS)" http://wiki.squid-cache.org/Features/HTTPS * curl Man Page - "Environment" https://curl.haxx.se/docs/manpage.html * Wget Manual - "Proxies" https://www.gnu.org/software/wget/manual/html_node/Proxies.html * Arch Linux Wiki - "Proxy settings" https://wiki.archlinux.org/index.php/proxy_settings Environment Variable Behavior ----------------------------- * `http_proxy` / `HTTP_PROXY` - A URL for a proxy server that will be used for any HTTP connections. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment variable takes precedence over the uppercase version. * `https_proxy` / `HTTPS_PROXY` - A URL for a proxy server that will be used for any HTTPS connections. This uses a TCP tunneling mode, meaning that the proxy server tunnels the encrypted socket connection through to the target and has no ability to intercept or unpack the contents of the connection. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment varible takes precedence over the uppercase version. * `no_proxy` / `NO_PROXY` - A comma-separated list of domain extensions that a proxy should *not* be used for. For example, if the value of `no_proxy` is `".example.com"` the a proxy should not be used for URLs ending with the `example.com` domain. The lowercase version of this environment variable takes precedence over the uppercase version. Bonus: ApiClient ---------------- In additional to supporting the above feature a refactoring of the habitat_http_client component led to the creation of an "API Client" struct to wrap up the underyling hyper HTTP behavior. It centers around a common base URL so that a proxy server can be determined and set up correctly for the target. As proxy authentication is implemented quite differently between plaintext and tunneled mode, it was easier to encapsulate this behavior inside the `ApiClient`. See the developer documentation for more usage details. Bonus: User-Agent Headers ------------------------- Finally, the wrapping of HTTP client calls behind the above mentioned `ApiClient` allows us to include a common `User-Agent` HTTP header which can be used to measure the various versions of software that connect to a Depot or other endpoints. The format of the `User-Agent` header value is similar to the analytics code implementation in the `hab` CLI. Final Word About depot-client API --------------------------------- Supporting an injection of a product name and version into each Depot client has led to the addition of function parameters in the codebase where a client is created. This isn't yet totally ideal, but future work is slated to reduce the number of Depot client creations and to simplfy its setup API. The author of this current feature asks for forgiveness in the meantime and not permission ;) Signed-off-by: Fletcher Nichol <[email protected]>
raskchanky
pushed a commit
that referenced
this pull request
Apr 16, 2019
This change re-introduces support for using HTTP proxies when contacting a Depot service. Previous versions of the hyper Rust crate were not sufficiently flexible to support both custom SSL contexts and proxy support which led to the removal of proxy support in #652. This new strategy adds support back by using a custom Hyper `NetworkConnector` that supports plaintext (i.e. http) and tunneled (i.e. https) proxying modes as well as optional basic authorization support. All 4 variants should cover the vast majority of corporate network/proxy setups. Background Reading Materials ---------------------------- Several references, documents, and RFCs were used in working up this feature and are shared here for context and for the curious: * RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1" https://tools.ietf.org/html/rfc2616 * RFC 2617 - "HTTP Authentication: Basic and Digest Access Authentication" https://tools.ietf.org/html/rfc2617 * RFC 2817 - "Upgrading to TLS Within HTTP/1.1" https://tools.ietf.org/html/rfc2817 * RFC draft - "Tunneling TCP based protocols through Web proxy servers" http://www.web-cache.com/Writings/Internet-Drafts/draft-luotonen-web-proxy-tunneling-01.txt * Polipo Manual - "Access Control" https://www.irif.univ-paris-diderot.fr/~jch/software/polipo/polipo.html#index-authCredentials * Squid Cache Wiki - "Feature: HTTPS (HTTP Secure or HTTP over SSL/TLS)" http://wiki.squid-cache.org/Features/HTTPS * curl Man Page - "Environment" https://curl.haxx.se/docs/manpage.html * Wget Manual - "Proxies" https://www.gnu.org/software/wget/manual/html_node/Proxies.html * Arch Linux Wiki - "Proxy settings" https://wiki.archlinux.org/index.php/proxy_settings Environment Variable Behavior ----------------------------- * `http_proxy` / `HTTP_PROXY` - A URL for a proxy server that will be used for any HTTP connections. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment variable takes precedence over the uppercase version. * `https_proxy` / `HTTPS_PROXY` - A URL for a proxy server that will be used for any HTTPS connections. This uses a TCP tunneling mode, meaning that the proxy server tunnels the encrypted socket connection through to the target and has no ability to intercept or unpack the contents of the connection. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment varible takes precedence over the uppercase version. * `no_proxy` / `NO_PROXY` - A comma-separated list of domain extensions that a proxy should *not* be used for. For example, if the value of `no_proxy` is `".example.com"` the a proxy should not be used for URLs ending with the `example.com` domain. The lowercase version of this environment variable takes precedence over the uppercase version. Bonus: ApiClient ---------------- In additional to supporting the above feature a refactoring of the habitat_http_client component led to the creation of an "API Client" struct to wrap up the underyling hyper HTTP behavior. It centers around a common base URL so that a proxy server can be determined and set up correctly for the target. As proxy authentication is implemented quite differently between plaintext and tunneled mode, it was easier to encapsulate this behavior inside the `ApiClient`. See the developer documentation for more usage details. Bonus: User-Agent Headers ------------------------- Finally, the wrapping of HTTP client calls behind the above mentioned `ApiClient` allows us to include a common `User-Agent` HTTP header which can be used to measure the various versions of software that connect to a Depot or other endpoints. The format of the `User-Agent` header value is similar to the analytics code implementation in the `hab` CLI. Final Word About depot-client API --------------------------------- Supporting an injection of a product name and version into each Depot client has led to the addition of function parameters in the codebase where a client is created. This isn't yet totally ideal, but future work is slated to reduce the number of Depot client creations and to simplfy its setup API. The author of this current feature asks for forgiveness in the meantime and not permission ;) Signed-off-by: Fletcher Nichol <[email protected]>
raskchanky
pushed a commit
that referenced
this pull request
Apr 16, 2019
…=fnichol [hab,hab-sup,hab-director] Add http & https proxy support. This change re-introduces support for using HTTP proxies when contacting a Depot service. Previous versions of the hyper Rust crate were not sufficiently flexible to support both custom SSL contexts and proxy support which led to the removal of proxy support in #652. This new strategy adds support back by using a custom Hyper `NetworkConnector` that supports plaintext (i.e. http) and tunneled (i.e. https) proxying modes as well as optional basic authorization support. All 4 variants should cover the vast majority of corporate network/proxy setups. Background Reading Materials ---------------------------- Several references, documents, and RFCs were used in working up this feature and are shared here for context and for the curious: * RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1" https://tools.ietf.org/html/rfc2616 * RFC 2617 - "HTTP Authentication: Basic and Digest Access Authentication" https://tools.ietf.org/html/rfc2617 * RFC 2817 - "Upgrading to TLS Within HTTP/1.1" https://tools.ietf.org/html/rfc2817 * RFC draft - "Tunneling TCP based protocols through Web proxy servers" http://www.web-cache.com/Writings/Internet-Drafts/draft-luotonen-web-proxy-tunneling-01.txt * Polipo Manual - "Access Control" https://www.irif.univ-paris-diderot.fr/~jch/software/polipo/polipo.html#index-authCredentials * Squid Cache Wiki - "Feature: HTTPS (HTTP Secure or HTTP over SSL/TLS)" http://wiki.squid-cache.org/Features/HTTPS * curl Man Page - "Environment" https://curl.haxx.se/docs/manpage.html * Wget Manual - "Proxies" https://www.gnu.org/software/wget/manual/html_node/Proxies.html * Arch Linux Wiki - "Proxy settings" https://wiki.archlinux.org/index.php/proxy_settings Environment Variable Behavior ----------------------------- * `http_proxy` / `HTTP_PROXY` - A URL for a proxy server that will be used for any HTTP connections. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment variable takes precedence over the uppercase version. * `https_proxy` / `HTTPS_PROXY` - A URL for a proxy server that will be used for any HTTPS connections. This uses a TCP tunneling mode, meaning that the proxy server tunnels the encrypted socket connection through to the target and has no ability to intercept or unpack the contents of the connection. You may specify a username and password through the proxy URL to support `Basic` authorization. For example: `"http://user:[email protected]:8123"`. The lowercase version of this environment varible takes precedence over the uppercase version. * `no_proxy` / `NO_PROXY` - A comma-separated list of domain extensions that a proxy should *not* be used for. For example, if the value of `no_proxy` is `".example.com"` the a proxy should not be used for URLs ending with the `example.com` domain. The lowercase version of this environment variable takes precedence over the uppercase version. Bonus: ApiClient ---------------- In additional to supporting the above feature a refactoring of the habitat_http_client component led to the creation of an "API Client" struct to wrap up the underyling hyper HTTP behavior. It centers around a common base URL so that a proxy server can be determined and set up correctly for the target. As proxy authentication is implemented quite differently between plaintext and tunneled mode, it was easier to encapsulate this behavior inside the `ApiClient`. See the developer documentation for more usage details. Bonus: User-Agent Headers ------------------------- Finally, the wrapping of HTTP client calls behind the above mentioned `ApiClient` allows us to include a common `User-Agent` HTTP header which can be used to measure the various versions of software that connect to a Depot or other endpoints. The format of the `User-Agent` header value is similar to the analytics code implementation in the `hab` CLI. Final Word About depot-client API --------------------------------- Supporting an injection of a product name and version into each Depot client has led to the addition of function parameters in the codebase where a client is created. This isn't yet totally ideal, but future work is slated to reduce the number of Depot client creations and to simplfy its setup API. The author of this current feature asks for forgiveness in the meantime and not permission ;) Signed-off-by: Fletcher Nichol <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[hab/sup] Find suitable cacerts for HTTP/SSL client calls.
This change is primarily driven by a desire to front a Depot with SSL, requiring that all API calls now require a client-side set of root certificates. The challenge with the Linux
habCLI is that it can't necessary rely on a consistent location to find acacert.pemor certificate directory. That is, it could be running on any number of 64-bit Linux operating systems, in minimal containers, etc. We are striving for a consistent user experience with as little setup as possible and ideally no extra wrangling with SSL certificates.The following is pulled from the
http-clientcomponent's developer documentation, but applies to the behavior ofhaband secondarily forhab-supandhab-directorat runtime.Linux Platforms Certificate Strategy
We need a set of root certificates when connected to SSL/TLS web endpoints and this usually boild down to using an all-in-one certificate file (such as a
cacert.pemfile) or a directory of files which are certificates. The strategy to location or use a reasonable set of certificates is as follows:SSL_CERT_FILEenvironment variable is set, then use its value for the certificates. Interally this is triggering default OpenSSL behavior for this environment variable. 2. If theSSL_CERT_DIRenvironment variable is set, then use its value for the directory containing certificates. Like theSSL_CERT_FILEcase above, this triggers default OpenSSL behavior for this environment variable.core/cacertsHabitat package is installed locally, then use the latest release'scacert.pemfile.cacert.pemwill be written in an SSL cache directory (by default/hab/cache/sslfor a root user and$HOME/.hab/cache/sslfor a non-root user) and this will be used. The contents of this file will be inlined in this crate at build time as a fallback insurance policy, meaning that if the a program using this code is operating in a minimal environment which may not contain system certificates, it can still operate. Once acore/cacertsHabitat package is present, the logic would fall back to preferring the package version to the cached/inline file version.Mac Platforms Certificate Strategy
The Mac platoform uses a Security Framework to store and find root certificates and the hyper library will default to using this on the Mac. Therefore the behavior on the Mac remains unchanged and will use the system's certificates.
Example In Minimal Linux Environment
Let's use a build of
core/hab-staticin a stock Alpine Linux Docker container which by default ships with no SSL root certificates (these are available by runningapk --update upgrade && apk add curl ca-certificates)First, we'll install
habonPATH, set an SSL Depot endpoint, and ensure that there is no cache (note that theRUST_LOGis set so we can see the resulting URLs and cacert loading logic):Now we'll install a package. Again, there are no SSL root certificates available to us (note that some DEBUG lines were removed for clarity):
You can see in the above output that we now have a
/hab/cache/ssl/cacert.pemwhich was used to make the Depot API calls. Now we'll install thecore/cacertspackage. As before,habwill usedthe cached
cacert.pemfile:Finally we'll install one last package, but this time
core/cacertsis installed and available, sohabwill use the certificates from that package instead, as they are most likely the same or newer than our inlined/cached version:[http-client] Drop support for http_proxy pending upstream support.
This change is admittedly a little sad, but is pragmatic in the short term. The current releases of the Hyper crate (which is the underlying HTTP client implementation) doesn't provide an API to construct a client with a particular SSL context and http_proxy support. As a secured Depot communication will trump most other features, we decided to disable http_proxy support until the team can get a change upstream to support this desired setup.
The previous commit (and therefore this one) will serve as the rough implementation of
new_hyper_clientwhen we can restore proxy support.