[test-failed]: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts - homepage app Welcome interstitial is displayed on a fresh on-prem install

[liza-mae]

Version: 7.14.1
Class: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts
Stack Trace:

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="homeWelcomeInterstitial"])
Wait timed out after 10057ms
   at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/basicGrp1/TASK/saas_run_kibana_tests/node/ess-testing/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
   at processTicksAndRejections (internal/process/task_queues.js:95:5) {
 remoteStacktrace: ''
}

Other test failures:

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/2223/testReport/

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[joshdover]

@liza-mae this test should be passing on Cloud now for the 7.14.2, 7.15.0, and 8.0.0 snapshots. The next 7.15.0 BC should have this resolved as well.

[liza-mae]

Version: 7.15.0
Class: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts
Stack Trace:

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="homeWelcomeInterstitial"])
Wait timed out after 10010ms
   at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/basicGrp1/TASK/saas_run_kibana_tests/node/ess-testing/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
   at processTicksAndRejections (internal/process/task_queues.js:95:5) {
 remoteStacktrace: ''
}

Other test failures:

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/2305/testReport/

[liza-mae]

This is still failing, can someone check why?

[pgayvallet]

@joshdover I don't see a PR associated with this issue. Do you remember what should have resolved this?

this test should be passing on Cloud now for the 7.14.2, 7.15.0, and 8.0.0 snapshots. The next 7.15.0 BC should have this resolved as well.

[joshdover]

I was hoping this would have fixed it: #110498

[liza-mae]

Version: 7.15.0
Class: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts
Stack Trace:

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="homeWelcomeInterstitial"])
Wait timed out after 10070ms
   at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/basicGrp1/TASK/saas_run_kibana_tests/node/ess-testing/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
   at processTicksAndRejections (internal/process/task_queues.js:95:5) {
 remoteStacktrace: ''
}

Other test failures:

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/2352/testReport/

[liza-mae]

Version: 7.15.1
Class: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts
Stack Trace:

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="homeWelcomeInterstitial"])
Wait timed out after 10112ms
   at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/basicGrp1/TASK/saas_run_kibana_tests/node/ess-testing/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
   at processTicksAndRejections (internal/process/task_queues.js:95:5) {
 remoteStacktrace: ''
}

Other test failures:

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/2458/testReport/

[liza-mae]

Version: 7.16.0
Class: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts
Stack Trace:

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="homeWelcomeInterstitial"])
Wait timed out after 10127ms
   at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/basicGrp1/TASK/saas_run_kibana_tests/node/ess-testing/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:895:17
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (node:internal/process/task_queues:96:5) {
 remoteStacktrace: ''
}

Other test failures:

• homepage app Welcome interstitial clicking on "Explore on my own" redirects to the "home" page

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/2471/testReport/

[liza-mae]

Version: 7.16.0
Class: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts
Stack Trace:

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="homeWelcomeInterstitial"])
Wait timed out after 10105ms
   at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/basicGrp1/TASK/saas_run_kibana_tests/node/ess-testing/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:895:17
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (node:internal/process/task_queues:96:5) {
 remoteStacktrace: ''
}

Other test failures:

• homepage app Welcome interstitial clicking on "Explore on my own" redirects to the "home" page

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/2636/testReport/

[liza-mae]

Version: 7.17.0
Class: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts
Stack Trace:

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="homeWelcomeInterstitial"])
Wait timed out after 10121ms
   at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/basicGrp1/TASK/saas_run_kibana_tests/node/ess-testing/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:898:17
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (node:internal/process/task_queues:96:5) {
 remoteStacktrace: ''
}

Other test failures:

• homepage app Welcome interstitial clicking on "Explore on my own" redirects to the "home" page

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/2783/testReport/

[liza-mae]

Version: 7.17.0
Class: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts
Stack Trace:

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="homeWelcomeInterstitial"])
Wait timed out after 10121ms
   at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/basicGrp1/TASK/saas_run_kibana_tests/node/ess-testing/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:901:17
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (node:internal/process/task_queues:96:5) {
 remoteStacktrace: ''
}

Other test failures:

• homepage app Welcome interstitial clicking on "Explore on my own" redirects to the "home" page

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/2872/testReport/

[liza-mae]

Version: 8.1.0
Class: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts
Stack Trace:

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="homeWelcomeInterstitial"])
Wait timed out after 10081ms
   at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/basicGrp1/TASK/saas_run_kibana_tests/node/ess-testing/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:901:17
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (node:internal/process/task_queues:96:5) {
 remoteStacktrace: ''
}

Other test failures:

• homepage app Welcome interstitial clicking on "Explore on my own" redirects to the "home" page

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/2876/testReport/

[joshdover]

@liza-mae Apologize for my ignorance here. How can I debug these or run these myself? Are there any screenshots being generated during failures?

I looked at https://github.com/elastic/elastic-stack-testing but I didn't see anything obviously relevant on how to work with these tests. Is there a better place to look?

[liza-mae]

@joshdover thanks for taking a look, I did a presentation on how to debug kibana cloud tests and that can be found on our drives Engineering -> ESTF -> Cloud Testing -> Kibana on Cloud, but it makes sense to put it in a README on https://github.com/elastic/elastic-stack-testing.

The test report is linked in the description of the failure (use the latest one). From there you can find the screenshots, I can post this one here:

[liza-mae]

Version: 8.1.0
Class: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts
Stack Trace:

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="homeWelcomeInterstitial"])
Wait timed out after 10094ms
   at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/basicGrp1/TASK/saas_run_kibana_tests/node/ess-testing/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:901:17
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (node:internal/process/task_queues:96:5) {
 remoteStacktrace: ''
}

Other test failures:

• homepage app Welcome interstitial clicking on "Explore on my own" redirects to the "home" page

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/2897/testReport/

[liza-mae]

Version: 7.17.1
Class: Chrome UI Functional Tests1.test/functional/apps/home/_welcome·ts
Stack Trace:

TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="homeWelcomeInterstitial"])
Wait timed out after 10065ms
   at /var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/basicGrp1/TASK/saas_run_kibana_tests/node/ess-testing/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/webdriver.js:901:17
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (node:internal/process/task_queues:96:5) {
 remoteStacktrace: ''
}

Other test failures:

• homepage app Welcome interstitial clicking on "Explore on my own" redirects to the "home" page

Test Report: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/2945/testReport/

[Dosant]

for 7.x everything seems to be working fine in cloud, so I think for 7.x tests are just flaky.

It seems like Enterprise Search is also adding a data stream by default that we should skip: logs-enterprise_search.api-default.
I validated that by creating another blank deployment but disabling Enterprise Search in that new deployment.

I confirmed this is the issue for 8.0+

[mattkime]

It seems like Enterprise Search is also adding a data stream by default that we should skip: logs-enterprise_search.api-default.

I'd really like to see this addressed via the system indices functionality. If its insufficient than we should figure out how to address it. We shouldn't have a growing list of 'things to ignore' in kibana.

[Dosant]

It seems that because #82851 is fixed in 8.1, that existing workaround with FLEET_ASSETS_TO_IGNORE is no longed needed because logs-* and metrics-* are no longer created by default.

So looks like the onboarding screen is only broken in 8.0.x because of logs-enterprise_search.api-default, but is working in 8.1.0. It seems like we are past the point where we would fix this in 8.0.x branch since 8.1 is released

However logs-enterprise_search.api-default slightly breaks the empty state of data views screen where it says, that "you have data, now create data view" because of it instead of "you don't have any data". But this is a less severe bug the missing welcome screen.

I propose the plan:

1. For main / 8.2:
   1.1 Remove additional handling of logs-* metrics-* in has_user_index_pattern since fleet no longer installs this data views by default
   1.2. Check if we can also get rid of specific handling in empty_prompts. Need to check with enterprise search team if they can make logs-enterprise_search.api-default hidden or installed when actually used?
2. For 8.1.1: add logs-enterprise_search.api-default to the existing logic
3. @afharo: unfortunately looks like core will still need to figure out flaky test. Just don't use 8.0 where the welcome screen is broken

[afharo]

@Dosant thanks for the investigation and explanation.

3. @afharo: unfortunately looks like core will still need to figure out flaky test. Just don't use 8.0 where the welcome screen is broken

Thanks for the heads up. I think we can wait for your other changes to be applied before looking further into this. AFAIK, these tests are not flaky on our "normal" CI. I wouldn't expect them to be flaky on Cloud.

Are there any issues we can follow to track your efforts for points 1 and 2?

[Dosant]

@afharo

Are there any issues we can follow to track your efforts for points 1 and 2?

small fix for logs-enterprise_search.api-default : #127787
No issue yet for larger effort of removing hardcoded indices or a strategy for an alternative. but here is related hasData services issue we are working on #124019

Thanks for the heads up. I think we can wait for your other changes to be applied before looking further into this. AFAIK, these tests are not flaky on our "normal" CI. I wouldn't expect them to be flaky on Cloud.

Most likely it is flaky because of some race condition of a data available in cloud cluster? Probably we could log existing non hidden indices / data streams on failure it those tests.

Btw, we have a dedicated test for has_user_index_pattern and I didn't see it fail in cloud:

https://github.com/elastic/kibana/blob/main/test/api_integration/apis/index_patterns/has_user_index_pattern/has_user_index_pattern.ts

[afharo]

Most likely it is flaky because of some race condition of a data available in cloud cluster? Probably we could log existing non hidden indices / data streams on failure it those tests.

IMO, the original flakiness was caused because we had a short timeout (500ms) so we didn't have the user waiting for too long to see the first content. @joshdover increased it to 10s (PR). It should have solved the flakiness.

However, I noticed we are explicitly disabling the welcome screen in the Cloud tests 😅 https://github.com/elastic/elastic-stack-testing/pull/1159 should remove that setting.

@liza-mae would you mind confirming that #127787 and https://github.com/elastic/elastic-stack-testing/pull/1159 fixed the bug for all versions?

Please, bear in mind that #127787 was merged after the last run according to the test history, and the backport to 8.1.1 is not merged yet.

[liza-mae]

Thanks @afharo I will kick off another build to catch these new changes, will let you know how it goes.

[liza-mae]

Confirmed fixed in latest 8.1.1 report - https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana-tests/3011/JOB=basicGrp1,TASK=saas_run_kibana_tests,node=ess-testing/testReport/