feat: support running kibana with different configs #1644

mdelapenya · 2021-10-06T06:03:56Z

What does this PR do?

This PRs does a few important things:

updates our kibana config file with the one at elastic-package's project
returns back to the configuration file instead of env vars for the Compose provider
renames the configurations folder to default, representing the name of the kibana profile to be used
adds a new test step that Bootstraps kibana with a profile. This new step can be used as a Background or in a Given clause (see docs)
creates an elastic-package profile on-the-fly for profiles different than default. It will copy there the contents of the profile's kibana.config.yml. Finally it will run the elastic-package tool with the new profile
adds a sample profile and feature file to verify this new feature

Why is it important?

It will allow running different scenarios with different Kibana configs.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests (make unit-test), and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

Author's Checklist

@nchaulet, please take a look at this PR, as it could be the foundation for your work.

How to test this PR locally

# using the compose provider
$ TAGS="policies && centos" TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE DEVELOPER_MODE=true ELASTIC_APM_ACTIVE=false  make -C e2e/_suites/fleet functional-test
# using the elastic-package provider
$ TAGS="policies && centos" TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE DEVELOPER_MODE=true ELASTIC_APM_ACTIVE=false PROVIDER="elastic-package"  make -C e2e/_suites/fleet functional-test

Related issues

Relates Add tests with preconfigured policies #1107

This will enable having multiple config files per test suite, using the profile abstraction: one dir per config with the profile name

It will represent the default profile

Default profile will be 'default'

The bootstrap method is called before the test suite (only once) AND before each test scenario, as it has been defined as a Background clause. If an scenario wants to configure a new config file for Kibana, a new profile file with the new kibana config must be created at the compose level, using the ProfileName as directory name.

elasticmachine · 2021-10-06T06:35:28Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-10-15T14:09:31.578+0000
Duration: 37 min 38 sec
Commit: b66afaa

Test stats 🧪

Test	Results
Failed	1
Passed	242
Skipped	0
Total	243

Test errors

Expand to view the tests failures

`Initializing / End-To-End Tests / windows-2019_fleet_fleet_mode_agent / Deploying the centos agent – Fleet Mode Agent`

Expand to view the error details

 Step system package dashboards are listed in Fleet

no stacktrace

Steps errors

Expand to view the steps failures

`Run functional tests for windows-2019:fleet:fleet_mode_agent && ~@nightly && ~debian && ~@skip:windo`

Took 15 min 10 sec . View more details here
Description: go test -timeout 60m -v --godog.format="junit:..\..\..\outputs\TEST-windows-fleet.xml" --godog.tags="fleet_mode_agent && ~@nightly && ~debian && ~@skip:windows && ~@skip:amd64"

`Archive the artifacts`

Took 0 min 0 sec . View more details here
Description: [2021-10-15T14:44:30.587Z] Archiving artifacts script returned exit code 1

`Archive the artifacts`

Took 0 min 1 sec . View more details here
Description: [2021-10-15T14:44:31.553Z] Archiving artifacts script returned exit code 1

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Initializing / End-To-End Tests / windows-2019_fleet_fleet_mode_agent / Deploying the centos agent – Fleet Mode Agent

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

mdelapenya · 2021-10-06T08:53:54Z

cli/config/compose/profiles/fleet/preconfigured-policies/kibana.config.yml

@@ -0,0 +1,20 @@
+---


@nchaulet here it's the config file as a placeholder for your config

mdelapenya · 2021-10-11T17:35:12Z

Not merging yet because I’ve started to see the following error when using elastic-package since today

[2021-10-11T16:06:25.239Z] time="2021-10-11T16:06:24Z" level=error msg="Error executing command" args="[run github.com/elastic/elastic-package stack up --daemon --verbose --version 8.0.0-2f7ca428-SNAPSHOT --services elasticsearch,fleet-server,kibana,package-registry -p default]" baseDir=. command=go env="map[kibanaDockerNamespace:kibana kibanaProfile:default kibanaVersion:8.0.0-2f7ca428-SNAPSHOT stackPlatform:linux/amd64 stackVersion:8.0.0-2f7ca428-SNAPSHOT]" error="exit status 1" stderr="missing $GOPATH\n"

I can reproduce it locally but only for this PR. Pinged @mtojek about that

mtojek · 2021-10-12T11:37:44Z

Taking a look

mtojek · 2021-10-12T11:44:58Z

I'm wondering if this problem isn't related to changes around Go 1.17.1 (cc @v1v).

mtojek · 2021-10-12T11:51:37Z

cli/config/compose/profiles/fleet/default/kibana.config.yml

-monitoring.ui.container.elasticsearch.enabled: true
-
-xpack.encryptedSavedObjects.encryptionKey: "12345678901234567890123456789012"
+xpack.monitoring.ui.container.elasticsearch.enabled: true


@mdelapenya feel free to modify the original config in the elastic-package tool.

v1v · 2021-10-12T12:00:25Z

/test

nchaulet · 2021-10-12T12:19:19Z

Thanks for adding this. I tried it locally and built a test on top of that and seems to work well for my use case

mtojek · 2021-10-12T14:32:07Z

The error about missing GOPATH may also come from go env: golang/go#43938 (comment) .

is there an option to print GOPATH and GOROOT?

mdelapenya · 2021-10-15T10:08:53Z

@mtojek it's possible to reproduce it locally from this PR too:

$ TAGS="fleet_mode_agent && centos && install" TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE DEVELOPER_MODE=true ELASTIC_APM_ACTIVE=false PROVIDER=elastic-package make -C e2e/_suites/fleet functional-test

I also tried bumping elastic-package to v0.22.0 with same results

mtojek · 2021-10-15T10:22:16Z

Is there an option to extract the elastic-package invocation from there? Might be easier to test.

EDIT:

I managed to reproduce it, investigating:

ERRO[2021-10-15T12:28:46+02:00] Error executing command                       args="[run github.com/elastic/elastic-package stack up --daemon --verbose --version 8.0.0-0da7fd6d-SNAPSHOT --services elasticsearch,fleet-server,kibana,package-registry -p default]" baseDir=. command=go env="map[kibanaDockerNamespace:kibana kibanaProfile:default kibanaVersion:8.0.0-0da7fd6d-SNAPSHOT stackPlatform:linux/amd64 stackVersion:8.0.0-0da7fd6d-SNAPSHOT]" error="exit status 1" stderr="missing $GOPATH\n"
FATA[2021-10-15T12:28:46+02:00] Could not bootstrap Fleet runtime dependencies  error="exit status 1"

EDIT2:

I can see these problems, am I missing any additional auth?

WARN[2021-10-15T12:31:21+02:00] An error ocurred while warming-up the Docker image. Continuing  error="Error response from daemon: pull access denied for docker.elastic.co/observability-ci/elastic-agent, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" image="docker.elastic.co/observability-ci/elastic-agent:8.0.0-0da7fd6d-SNAPSHOT"
WARN[2021-10-15T12:31:25+02:00] An error ocurred while warming-up the Docker image. Continuing  error="Error response from daemon: pull access denied for docker.elastic.co/observability-ci/elastic-agent-ubi8, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" image="docker.elastic.co/observability-ci/elastic-agent-ubi8:8.0.0-0da7fd6d-SNAPSHOT"
WARN[2021-10-15T12:31:27+02:00] An error ocurred while warming-up the Docker image. Continuing  error="Error response from daemon: pull access denied for docker.elastic.co/observability-ci/elasticsearch, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" image="docker.elastic.co/observability-ci/elasticsearch:8.0.0-0da7fd6d-SNAPSHOT"
WARN[2021-10-15T12:31:30+02:00] An error ocurred while warming-up the Docker image. Continuing  error="Error response from daemon: pull access denied for docker.elastic.co/observability-ci/kibana, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" image="docker.elastic.co/observability-ci/kibana:8.0.0-0da7fd6d-SNAPSHOT"

mtojek · 2021-10-15T10:52:08Z

Ok, so the problem is with shell executor in e2e-testing. I sneaked small change into the ./internal/shell/shell.go:

cmd.Env = append(cmd.Env, "GOPATH=/Users/marcin.tojek/go")

and it fails with different error:

ERRO[2021-10-15T12:50:15+02:00] Error executing command                       args="[run github.com/elastic/elastic-package stack up --daemon --verbose --version 8.0.0-0da7fd6d-SNAPSHOT --services elasticsearch,fleet-server,kibana,package-registry -p default]" baseDir=. command=go env="map[kibanaDockerNamespace:kibana kibanaProfile:default kibanaVersion:8.0.0-0da7fd6d-SNAPSHOT stackPlatform:linux/amd64 stackVersion:8.0.0-0da7fd6d-SNAPSHOT]" error="exit status 1" stderr="build cache is required, but could not be located: GOCACHE is not defined and $HOME is not defined\n"

mdelapenya · 2021-10-15T11:00:45Z

Ok, so the problem is with shell executor in e2e-testing. I sneaked small change into the ./internal/shell/shell.go:

cmd.Env = append(cmd.Env, "GOPATH=/Users/marcin.tojek/go")

and it fails with different error:

ERRO[2021-10-15T12:50:15+02:00] Error executing command                       args="[run github.com/elastic/elastic-package stack up --daemon --verbose --version 8.0.0-0da7fd6d-SNAPSHOT --services elasticsearch,fleet-server,kibana,package-registry -p default]" baseDir=. command=go env="map[kibanaDockerNamespace:kibana kibanaProfile:default kibanaVersion:8.0.0-0da7fd6d-SNAPSHOT stackPlatform:linux/amd64 stackVersion:8.0.0-0da7fd6d-SNAPSHOT]" error="exit status 1" stderr="build cache is required, but could not be located: GOCACHE is not defined and $HOME is not defined\n"

Mmm you are right. I think I have the fix for it, as we are overriding the entire ENV for cmd.Exec instead of appending!

Good catch!

Instead of replacing it, we want to append!

mdelapenya · 2021-10-15T14:08:59Z

Merging, as the test error comes from the windows issue with fleet

mdelapenya · 2021-10-15T15:52:09Z

@nchaulet feel free to start building on top! 🚀

* chore: match elastic-package's kibana config * chore: bring back kibana's configuration file This will enable having multiple config files per test suite, using the profile abstraction: one dir per config with the profile name * chore: rename configurations to default It will represent the default profile * chore: make kibana profile configurable Default profile will be 'default' * feat: support bootstrapping the stack with a new kibana config The bootstrap method is called before the test suite (only once) AND before each test scenario, as it has been defined as a Background clause. If an scenario wants to configure a new config file for Kibana, a new profile file with the new kibana config must be created at the compose level, using the ProfileName as directory name. * feat: create an on-the-fly elastic-package's profile from compose * chore: override default kibana config for elastic-package * fix: remove unused code * docs: document kibana config and profiles * chore: add a profile for preconfgured-policies * elas * chore: add placeholder for preconfigured policies * chore: support passing custom kibana ref to elastic-package * fix: honour current env Instead of replacing it, we want to append! * chore: unit test that we are not overriding the env (cherry picked from commit a207500)

* chore: match elastic-package's kibana config * chore: bring back kibana's configuration file This will enable having multiple config files per test suite, using the profile abstraction: one dir per config with the profile name * chore: rename configurations to default It will represent the default profile * chore: make kibana profile configurable Default profile will be 'default' * feat: support bootstrapping the stack with a new kibana config The bootstrap method is called before the test suite (only once) AND before each test scenario, as it has been defined as a Background clause. If an scenario wants to configure a new config file for Kibana, a new profile file with the new kibana config must be created at the compose level, using the ProfileName as directory name. * feat: create an on-the-fly elastic-package's profile from compose * chore: override default kibana config for elastic-package * fix: remove unused code * docs: document kibana config and profiles * chore: add a profile for preconfgured-policies * elas * chore: add placeholder for preconfigured policies * chore: support passing custom kibana ref to elastic-package * fix: honour current env Instead of replacing it, we want to append! * chore: unit test that we are not overriding the env (cherry picked from commit a207500) Co-authored-by: Manuel de la Peña <[email protected]>

mdelapenya added 12 commits October 6, 2021 04:55

chore: match elastic-package's kibana config

bc97f9e

chore: bring back kibana's configuration file

c46b403

This will enable having multiple config files per test suite, using the profile abstraction: one dir per config with the profile name

chore: rename configurations to default

15ed9c3

It will represent the default profile

chore: make kibana profile configurable

c839749

Default profile will be 'default'

feat: create an on-the-fly elastic-package's profile from compose

433623a

chore: override default kibana config for elastic-package

95f1591

fix: remove unused code

d942746

docs: document kibana config and profiles

51cd70a

chore: add a profile for preconfgured-policies

e260cf3

elas

1fdf0af

chore: add placeholder for preconfigured policies

f949d27

mdelapenya self-assigned this Oct 6, 2021

mdelapenya requested a review from a team October 6, 2021 06:03

mdelapenya mentioned this pull request Oct 6, 2021

Add tests with preconfigured policies #1107

Closed

cachedout approved these changes Oct 6, 2021

View reviewed changes

mdelapenya marked this pull request as ready for review October 6, 2021 08:53

mdelapenya commented Oct 6, 2021

View reviewed changes

mdelapenya requested a review from adam-stokes October 6, 2021 09:23

chore: support passing custom kibana ref to elastic-package

475b9bd

adam-stokes approved these changes Oct 11, 2021

View reviewed changes

mtojek reviewed Oct 12, 2021

View reviewed changes

Merge branch 'master' into configurable-kibana-config

e3ba427

mdelapenya added 2 commits October 15, 2021 13:01

fix: honour current env

82e4ee3

Instead of replacing it, we want to append!

chore: unit test that we are not overriding the env

249999d

Merge branch 'master' into configurable-kibana-config

b66afaa

mdelapenya added v7.14.0 v7.15.0 v7.16.0 labels Oct 15, 2021

mdelapenya merged commit a207500 into elastic:master Oct 15, 2021

mergify bot mentioned this pull request Oct 15, 2021

[7.x](backport #1644) feat: support running kibana with different configs #1660

Merged

mergify bot mentioned this pull request Oct 15, 2021

[7.15.x](backport #1644) feat: support running kibana with different configs #1661

Merged

mergify bot mentioned this pull request Oct 15, 2021

[7.14.x](backport #1644) feat: support running kibana with different configs #1662

Merged

mdelapenya mentioned this pull request Oct 26, 2021

chore: bump elastic-package to v0.25.0 #1721

Merged

8 tasks

mdelapenya mentioned this pull request Nov 4, 2021

fix: remove preconfigured policies from 7.15 #1774

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support running kibana with different configs #1644

feat: support running kibana with different configs #1644

mdelapenya commented Oct 6, 2021

elasticmachine commented Oct 6, 2021 •

edited

Loading

Build stats

Test stats 🧪

`Initializing / End-To-End Tests / windows-2019_fleet_fleet_mode_agent / Deploying the centos agent – Fleet Mode Agent`

`Run functional tests for windows-2019:fleet:fleet_mode_agent && ~@nightly && ~debian && ~@skip:windo`

`Archive the artifacts`

`Archive the artifacts`

Genuine test errors

mdelapenya Oct 6, 2021

mdelapenya commented Oct 11, 2021

mtojek commented Oct 12, 2021

mtojek commented Oct 12, 2021

mtojek Oct 12, 2021

v1v commented Oct 12, 2021

nchaulet commented Oct 12, 2021

mtojek commented Oct 12, 2021 •

edited

Loading

mdelapenya commented Oct 15, 2021

mtojek commented Oct 15, 2021 •

edited

Loading

mtojek commented Oct 15, 2021

mdelapenya commented Oct 15, 2021

mdelapenya commented Oct 15, 2021

mdelapenya commented Oct 15, 2021

feat: support running kibana with different configs #1644

feat: support running kibana with different configs #1644

Conversation

mdelapenya commented Oct 6, 2021

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

elasticmachine commented Oct 6, 2021 • edited Loading

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Initializing / End-To-End Tests / windows-2019_fleet_fleet_mode_agent / Deploying the centos agent – Fleet Mode Agent

Steps errors

Run functional tests for windows-2019:fleet:fleet_mode_agent && ~@nightly && ~debian && ~@skip:windo

Archive the artifacts

Archive the artifacts

🐛 Flaky test report

Genuine test errors

🤖 GitHub comments

mdelapenya Oct 6, 2021

Choose a reason for hiding this comment

mdelapenya commented Oct 11, 2021

mtojek commented Oct 12, 2021

mtojek commented Oct 12, 2021

mtojek Oct 12, 2021

Choose a reason for hiding this comment

v1v commented Oct 12, 2021

nchaulet commented Oct 12, 2021

mtojek commented Oct 12, 2021 • edited Loading

mdelapenya commented Oct 15, 2021

mtojek commented Oct 15, 2021 • edited Loading

mtojek commented Oct 15, 2021

mdelapenya commented Oct 15, 2021

mdelapenya commented Oct 15, 2021

mdelapenya commented Oct 15, 2021

elasticmachine commented Oct 6, 2021 •

edited

Loading

`Initializing / End-To-End Tests / windows-2019_fleet_fleet_mode_agent / Deploying the centos agent – Fleet Mode Agent`

`Run functional tests for windows-2019:fleet:fleet_mode_agent && ~@nightly && ~debian && ~@skip:windo`

`Archive the artifacts`

`Archive the artifacts`

mtojek commented Oct 12, 2021 •

edited

Loading

mtojek commented Oct 15, 2021 •

edited

Loading