[Ingest Manager] Use optional registryProxyUrl setting when contacting Registry

[jfsiii]

Summary

If given a xpack.fleet.registryProxyUrl setting, Package Manager will use it when contacting the Registry. This only affects the outbound connection Package Manager makes to the Registry to search for available packages, download assets, etc.

Configuration

Initial PR: common environment variables

Currently the value must come from a list of popular environment variables which include ALL_PROXY, HTTPS_PROXY, lowercase versions of those, and many more.

Start kibana with a proxy set in an environment variable like: HTTPS_PROXY=https://localhost:8443 yarn start

update based on discussion in the comments, the initial environment variables approach was removed in favor of xpack.ingestManager.registryProxyUrl

see #78968 for additional configuration coming later

Checklist

• Documentation was added for features that require explanation or tutorials. Created [Ingest Manager] Document proxy feature from #78648 #78961 to track
• Unit or functional tests were updated or added to match the most common scenarios

Created #78968 to track the additional configuration work

refs #70710

[ruflin]

Change LGTM. One part I was wondering is we need to support an HTTP Proxy?

Happy to get this in as is as the most important part first is that it does not break any existing setup. As long as no environment variables are set, it seems to be the exact same options as before so we should be fine here.

We need to follow up with the config options in the yaml file and docs on this.

@EricDavisX Could you help test this as soon as it got merged.

[ph]

@jfsiii We should document that feature.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[jfsiii]

@elasticmachine merge upstream

[jfsiii]

@elasticmachine merge upstream

[jfsiii]

@ph I'd like to merge this as-is so it's available as a snapshot for people to try. I updated the summary and created #78961 for additional documentation.

What release_note label should I use here?

[jfsiii]

ping @spalger for package.json change

[jen-huang]

@jfsiii I would use release_note:enhancement and add a Release note section to the PR description if you'd like to provide more information, otherwise the release note script will simply use your PR title: process doc.

If you'd like to take a stab at writing public docs, you can open a PR in observability-docs or file an issue in there.

[spalger]

package.json changes LGTM

[jfsiii]

@elasticmachine merge upstream

[ph]

@kobelb Just to clarify the intend of you comment, you are suggesting to block this change and that we do no allow users to use environment variables and instead make an explicit choice to use configuration options in the yaml instead?

This seems weird to me, just to the nature of _HTTP(S)PROXY variables and expectations. WhereI think the expectation would be the following.

• If a standard environment proxy variable is present uses it.
• If an explicit option is configured in either the Elasticsearch host or EPR would be to override it.

This would align with other Elastic products like beats.

[kobelb]

@kobelb Just to clarify the intend of you comment, you are suggesting to block this change and that we do no allow users to use environment variables and instead make an explicit choice to use configuration options in the yaml instead?

Correct.

This seems weird to me, just to the nature of _HTTP(S)PROXY variables and expectations. WhereI think the expectation would be the following.

If a standard environment proxy variable is present uses it.
If an explicit option is configured in either the Elasticsearch host or EPR would be to override it.
This would align with other Elastic products like beats.

So you're suggesting that when the HTTP(S)_PROXY environment variable is set, that Kibana should intelligently know when the defaults for the various servers are on the public internet and respect it appropriately. For example, it wouldn't use the HTTP(S)_PROXY environment variable when communication with Elasticsearch, as it's by default at http://localhost:9200 and we're rather confident the user isn't doing something weird with DNS to make this resolve to an IP address on the public internet. And in situations like EPR where we are confident that the server we're communicating with is on the public internet, the HTTP(S)_PROXY would be automatically respected? Is this roughly what Beats is doing?

[jfsiii]

Are end-users able to configure the URL of the Elastic Package Registry using a kibana.yml setting? If they aren't able to currently, do we intend for them to be able to do so in the future?

Yes they can via set it via xpack.ingestManager.registryUrl. However, it is an undocumented & officially unsupported flag and currently requires Gold+

I'd like to highlight that this PR only adds the proxy information to the outbound request from Kibana server to the Registry server. Communication with ES is via the Saved Object & legacy ES clients from the core context and is unaffected.

In my view, we're trying to be good citizens and respect the intentional & low-level settings the user has made. If the user sets a *_PROXY value, they know what to expect. Moreover, so many other applications, libraries, etc support these variables that we surprise them by not picking them up.

If the user sets both a proxy URL and a registry URL, but doesn't want the registry request to be proxied, there are ways to specify that using the environment variables like NO_PROXY

The original issue #70710 shows the user doing something like this by combining a few of the variables together to specify "All requests (except this list) should go through proxy"

        - name: kibana
          env:
          - name: HTTP_PROXY
            value: http://***:3128
          - name: HTTPS_PROXY
            value: http://*****:3128
          - name: NO_PROXY
            value: 172.20.0.1:443,169.254.169.254,.cluster.local

We use proxy-from-env, also used in the CLI, to compare the given URL (the registry in our case) to the rules given via the many possible environment variables and Do The Right Thing ™️

The library has a nice test suite which covers a host of combinations.

[jfsiii]

@kobelb Our posts crossed but I think I answered your questions with mine. Just in case:

So you're suggesting that when the HTTP(S)_PROXY environment variable is set, that Kibana should intelligently know when the defaults for the various servers are on the public internet and respect it appropriately. For example, it wouldn't use the HTTP(S)_PROXY environment variable when communication with Elasticsearch, as it's by default at http://localhost:9200 and we're rather confident the user isn't doing something weird with DNS to make this resolve to an IP address on the public internet.

This PR only adds the proxy info for Kibana's outbound request to EPR. None of the communication with ES is affected.

And in situations like EPR where we are confident that the server we're communicating with is on the public internet, the HTTP(S)_PROXY would be automatically respected?

only requests to EPR are involved. The function makes no distinction between public/private internet. Merely "does given URL need to be proxied based on these system-level rules". The interaction between the variables isn't a formal spec but it's a strong convention and widely supported. proxy-from-env does an excellent job of encapsulating all that logic.

[kobelb]

I was previously unaware of the behavior of the NO_PROXY environment variable, that does change some of my concerns. Thanks for bringing that to my attention @jfsiii.

However, there is a de-facto standard within Kibana that proxy servers should be configured using kibana.yml settings. This is the approach that has been implemented by both Reporting and Actions. In order to provide a consistent user-experience for our users, I'd highly recommend that Ingest Manager follow suit and add kibana.yml settings as well.

I do think it's worth-while to have a broader discussion about whether or not Kibana, and potentially the entire Stack, should respect the HTTP(S)_PROXY and related environment variables. However, this would have to be implemented throughout Kibana in a major version upgrade, as it's a breaking change.

It's worth noting that ES does not support these environment variables because they are not standard for the Java ecosystem. There's some discussion about this here. We've recently changed our attitude and no longer expect users to know that Elasticsearch is built in Java, or that Kibana is built in Node, so it's worth re-evaluating the justification.

[ph]

Looking at the discussion, as much as I would like to fix that quickly for our users, I do understand the concerns of @kobelb for changing the behavior like this, even if I think its was right behavior to do. I would prefer we do not change the behavior of kibana.

@jfsiii Can you look into adding settings in the kibana.yml file to support that behavior and remove environment handling?

@kobelb Concerning the environment variables, I think this fall into platform and should be tracked on your side. WDYT?

[jfsiii]

@ph I totally agree and remaining consistent (config values) was always the plan #78968. This PR used the env variables in an attempt to merge something that would allow testing/feedback ahead of FF while we sorted out the config name and any other items. That window has closed and FF is tomorrow, so I'll add the config value and resubmit.

I'll update the description and #78968 but I'm going to use registryProxy* vs .proxy*

[jfsiii]

@elasticmachine merge upstream

[nchaulet]

👍

[jfsiii]

@kobelb I swapped env variables for config in d91456d

[jen-huang]

Could we remove the dependency changes now that we don't need proxy-from-env? Otherwise code LGTM

[ph]

LGTM, lets add it to the documentation.

[jfsiii]

This will fail until #79724 is in main. Will update ASAP

[jfsiii]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 9aa3b25

Metrics [docs]

distributable file count

id	before	after	diff
default	48446	48447	+1

History

• 💔 Build #80183 failed 87846e7
• 💔 Build #80156 failed 1874d39
• 💚 Build #80138 succeeded 615886d
• 💚 Build #79802 succeeded d91456d
• 💚 Build #79682 succeeded 2124232

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream