[Obs AI Assistant] Unskip user instructions test (#205656) #205897

viduni94 · 2025-01-08T13:47:26Z

Closes #205656

Summary

Problem

There seems to be a race condition and state contamination between tests in the KB user instructions tests.

Solution

Clear knowledge base and conversations after tests
Remove title interceptor --> we don't need title interception for these tests

Checklist

Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

kibanamachine · 2025-01-08T20:52:27Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7670

[❌] x-pack/test_serverless/api_integration/test_suites/observability/ai_assistant/config.ts: 160/200 tests passed.

see run history

kibanamachine · 2025-01-08T22:33:46Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7674

[✅] x-pack/test_serverless/api_integration/test_suites/observability/ai_assistant/config.ts: 100/100 tests passed.

see run history

kibanamachine · 2025-01-09T01:16:58Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7675

[❌] x-pack/test_serverless/api_integration/test_suites/observability/ai_assistant/config.ts: 175/200 tests passed.

see run history

kibanamachine · 2025-01-09T03:04:46Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7676

[❌] x-pack/test/observability_ai_assistant_api_integration/enterprise/config.ts: 174/200 tests passed.

see run history

elasticmachine · 2025-01-09T13:08:27Z

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

…t the title (elastic#205656)

github-actions · 2025-01-09T13:10:30Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

arturoliduena · 2025-01-09T13:50:53Z

...y_ai_assistant_api_integration/tests/knowledge_base/knowledge_base_user_instructions.spec.ts

-            .interceptConversation({ name: 'conversation', response: 'I, the LLM, hear you!' })
-            .completeAfterIntercept(),
-        ];
+        const interceptPromise = proxy


It does not wait for the LLM-generated title to be generated. if this is intentional?

Yes, we don't check for the title in the test. Therefore, I removed the title interception

kibanamachine · 2025-01-09T14:07:40Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7682

[❌] x-pack/test/observability_ai_assistant_api_integration/enterprise/config.ts: 74/100 tests passed.

see run history

kibanamachine · 2025-01-09T14:20:06Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7681

[✅] x-pack/test_serverless/api_integration/test_suites/observability/ai_assistant/config.ts: 100/100 tests passed.

see run history

sorenlouv · 2025-01-09T14:36:04Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7682

[❌] x-pack/test/observability_ai_assistant_api_integration/enterprise/config.ts: 74/100 tests passed.

see run history

Any idea why the first flaky test runner failed and the second passed? Looks like no code changes between the two.

viduni94 · 2025-01-09T14:41:10Z

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7682

[❌] x-pack/test/observability_ai_assistant_api_integration/enterprise/config.ts: 74/100 tests passed.
see run history

Any idea why the first flaky test runner failed and the second passed? Looks like no code changes between the two.

Hey @sorenlouv
One was for serverless, one was for stateful.

Serverless tests are not flaky anymore
In stateful tests, the knowledge base migration test is flaky - Failing test: Observability AI Assistant API Integration tests (enterprise).x-pack/test/observability_ai_assistant_api_integration/tests/knowledge_base/knowledge_base_migration·spec·ts - Observability AI Assistant API tests knowledge_base/knowledge_base_migration.spec.ts When there are knowledge base entries (from 8.15 or earlier) that does not contain semantic_text embeddings after migrating the docs have semantic_text embeddings #202926

elasticmachine · 2025-01-09T15:01:06Z

💚 Build Succeeded

Buildkite Build
Commit: 666c8f1

Metrics [docs]

✅ unchanged

cc @viduni94

kibanamachine · 2025-01-09T15:02:23Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12692534725

…astic#205897) Closes elastic#205656 ## Summary ### Problem There seems to be a race condition and state contamination between tests in the KB user instructions tests. ### Solution - Clear knowledge base and conversations after tests - Remove title interceptor --> we don't need title interception for these tests ### Checklist - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit 37d7a5e)

kibanamachine · 2025-01-09T15:07:35Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…astic#205897) Closes elastic#205656 ## Summary ### Problem There seems to be a race condition and state contamination between tests in the KB user instructions tests. ### Solution - Clear knowledge base and conversations after tests - Remove title interceptor --> we don't need title interception for these tests ### Checklist - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

kibanamachine · 2025-01-10T15:49:13Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

…astic#205897) Closes elastic#205656 ## Summary ### Problem There seems to be a race condition and state contamination between tests in the KB user instructions tests. ### Solution - Clear knowledge base and conversations after tests - Remove title interceptor --> we don't need title interception for these tests ### Checklist - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit 37d7a5e)

…5897) (#206082) # Backport This will backport the following commits from `main` to `8.x`: - [[Obs AI Assistant] Unskip user instructions test (#205656) (#205897)](#205897)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  we don't need title interception for\r\nthese tests\r\n\r\n### Checklist\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_note:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"37d7a5efb760dc271a4fb26f1eeba0d66e037b07","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Obs AI Assistant","ci:project-deploy-observability","backport:version","v8.18.0"],"title":"[Obs AI Assistant] Unskip user instructions test (#205656)","number":205897,"url":"https://github.com/elastic/kibana/pull/205897","mergeCommit":{"message":"[Obs AI Assistant] Unskip user instructions test (#205656) (#205897)\n\nCloses https://github.com/elastic/kibana/issues/205656\r\n\r\n## Summary\r\n\r\n### Problem\r\nThere seems to be a race condition and state contamination between tests\r\nin the KB user instructions tests.\r\n\r\n### Solution\r\n- Clear knowledge base and conversations after tests\r\n- Remove title interceptor --> we don't need title interception for\r\nthese tests\r\n\r\n### Checklist\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_note:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"37d7a5efb760dc271a4fb26f1eeba0d66e037b07"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/205897","number":205897,"mergeCommit":{"message":"[Obs AI Assistant] Unskip user instructions test (#205656) (#205897)\n\nCloses https://github.com/elastic/kibana/issues/205656\r\n\r\n## Summary\r\n\r\n### Problem\r\nThere seems to be a race condition and state contamination between tests\r\nin the KB user instructions tests.\r\n\r\n### Solution\r\n- Clear knowledge base and conversations after tests\r\n- Remove title interceptor --> we don't need title interception for\r\nthese tests\r\n\r\n### Checklist\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_note:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"37d7a5efb760dc271a4fb26f1eeba0d66e037b07"}},{"branch":"8.x","label":"v8.18.0","branchLabelMappingKey":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Viduni Wickramarachchi <[email protected]>

…astic#205897) Closes elastic#205656 ## Summary ### Problem There seems to be a race condition and state contamination between tests in the KB user instructions tests. ### Solution - Clear knowledge base and conversations after tests - Remove title interceptor --> we don't need title interception for these tests ### Checklist - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

viduni94 added release_note:skip Skip the PR/issue when compiling release notes Team:Obs AI Assistant Observability AI Assistant backport:version Backport to applied version labels v8.18.0 labels Jan 8, 2025

viduni94 self-assigned this Jan 8, 2025

viduni94 marked this pull request as ready for review January 9, 2025 13:08

viduni94 requested a review from a team as a code owner January 9, 2025 13:08

viduni94 added 6 commits January 9, 2025 08:10

[Obs AI Assistant] Unskip user instructions test (elastic#205656)

c14f85e

[Obs AI Assistant] Update getConversationForUser to properly intercep…

05de942

…t the title (elastic#205656)

[Obs AI Assistant] Unskip user instructions test (elastic#205656)

ebd6775

[Obs AI Assistant] Unskip user instructions test (elastic#205656)

135f53e

[Obs AI Assistant] Remove title interceptor (elastic#205656)

075a4b4

[Obs AI Assistant] Unskip user instructions test (elastic#205656)

666c8f1

viduni94 force-pushed the kb-user-instructions-flaky-test branch from 313bb9e to 666c8f1 Compare January 9, 2025 13:10

botelastic bot added the ci:project-deploy-observability Create an Observability project label Jan 9, 2025

elastic deleted a comment from kibanamachine Jan 9, 2025

arturoliduena reviewed Jan 9, 2025

View reviewed changes

sorenlouv approved these changes Jan 9, 2025

View reviewed changes

viduni94 merged commit 37d7a5e into elastic:main Jan 9, 2025
10 checks passed

kibanamachine added the v9.0.0 label Jan 9, 2025

kibanamachine mentioned this pull request Jan 9, 2025

[8.x] [Obs AI Assistant] Unskip user instructions test (#205656) (#205897) #206082

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jan 10, 2025

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jan 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Obs AI Assistant] Unskip user instructions test (#205656) #205897

[Obs AI Assistant] Unskip user instructions test (#205656) #205897

viduni94 commented Jan 8, 2025 •

edited by kibanamachine

Loading

kibanamachine commented Jan 8, 2025

kibanamachine commented Jan 8, 2025

kibanamachine commented Jan 9, 2025

kibanamachine commented Jan 9, 2025

elasticmachine commented Jan 9, 2025

github-actions bot commented Jan 9, 2025

arturoliduena Jan 9, 2025

viduni94 Jan 9, 2025 •

edited

Loading

kibanamachine commented Jan 9, 2025

kibanamachine commented Jan 9, 2025

sorenlouv commented Jan 9, 2025

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7682

viduni94 commented Jan 9, 2025 •

edited

Loading

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7682

elasticmachine commented Jan 9, 2025

kibanamachine commented Jan 9, 2025

kibanamachine commented Jan 9, 2025

kibanamachine commented Jan 10, 2025

[Obs AI Assistant] Unskip user instructions test (#205656) #205897

[Obs AI Assistant] Unskip user instructions test (#205656) #205897

Conversation

viduni94 commented Jan 8, 2025 • edited by kibanamachine Loading

Summary

Problem

Solution

Checklist

kibanamachine commented Jan 8, 2025

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7670

kibanamachine commented Jan 8, 2025

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7674

kibanamachine commented Jan 9, 2025

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7675

kibanamachine commented Jan 9, 2025

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7676

elasticmachine commented Jan 9, 2025

github-actions bot commented Jan 9, 2025

🤖 GitHub comments

arturoliduena Jan 9, 2025

Choose a reason for hiding this comment

viduni94 Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

kibanamachine commented Jan 9, 2025

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7682

kibanamachine commented Jan 9, 2025

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7681

sorenlouv commented Jan 9, 2025

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7682

viduni94 commented Jan 9, 2025 • edited Loading

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7682

elasticmachine commented Jan 9, 2025

💚 Build Succeeded

Metrics [docs]

kibanamachine commented Jan 9, 2025

kibanamachine commented Jan 9, 2025

💚 All backports created successfully

Questions ?

kibanamachine commented Jan 10, 2025

viduni94 commented Jan 8, 2025 •

edited by kibanamachine

Loading

viduni94 Jan 9, 2025 •

edited

Loading

viduni94 commented Jan 9, 2025 •

edited

Loading