Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/prebuilt_rules/fleet_integration·ts - detection engine api security and spaces enabled - Prebuilt Rules install_prebuilt_rules_from_real_package should install prebuilt rules from the package storage #167056

Closed
Tracked by #165359
kibanamachine opened this issue Sep 22, 2023 · 9 comments · Fixed by #169791
Closed
Tracked by #165359

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/prebuilt_rules/fleet_integration·ts - detection engine api security and spaces enabled - Prebuilt Rules install_prebuilt_rules_from_real_package should install prebuilt rules from the package storage #167056

kibanamachine opened this issue Sep 22, 2023 · 9 comments · Fixed by #169791
Assignees
@jpdjere
Labels
8.12 candidate failed-test A test failure on a tracked branch, potentially flaky-test Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Sep 22, 2023 •
edited
Loading

A test failed on a tracked branch

JestAssertionError: expect(received).toBeGreaterThan(expected)

Expected: > 0
Received:   0
    at Context.<anonymous> (fleet_integration.ts:53:66)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (wrap_function.js:73:16) {
  matcherResult: {
    message: '\x1B[2mexpect(\x1B[22m\x1B[31mreceived\x1B[39m\x1B[2m).\x1B[22mtoBeGreaterThan\x1B[2m(\x1B[22m\x1B[32mexpected\x1B[39m\x1B[2m)\x1B[22m\n' +
      '\n' +
      'Expected: > \x1B[32m0\x1B[39m\n' +
      'Received:   \x1B[31m0\x1B[39m',
    pass: false
  }
}

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Sep 22, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Sep 22, 2023
@kibanamachine kibanamachine added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 22, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Sep 22, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@MadameSheema MadameSheema added Team:Detections and Resp Security Detection Response Team Team:Detection Rule Management Security Detection Rule Management Team labels Oct 17, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

mistic added a commit that referenced this issue Oct 17, 2023
@mistic
skip flaky suite (#167056)
514d99e
@mistic
Copy link
Member

mistic commented Oct 17, 2023

Skipped.

main: 514d99e

@banderror banderror added 8.12 candidate Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area labels Oct 18, 2023
@banderror banderror assigned jpdjere and unassigned banderror Oct 18, 2023
@banderror banderror mentioned this issue Oct 18, 2023
Closed
hop-dev pushed a commit to hop-dev/kibana that referenced this issue Oct 18, 2023
@mistic @hop-dev
skip flaky suite (elastic#167056)
7bdbb07
@jpdjere jpdjere mentioned this issue Oct 25, 2023
Merged
maximpn pushed a commit that referenced this issue Oct 27, 2023
@jpdjere
[Security Solution ] Fix flake in `/prebuilt_rules/fleet_integration.…
8bf2931 
…ts` API Integration test (#169791)

Fixes: #167056

## Summary

- Flake cannot be reproduced on 500 runs on flaky test runner.
- Latest fail points to [a `socket timeout` error](https://buildkite.com/elastic/kibana-on-merge/builds/36893#018b3d21-52ea-40a5-92a2-24cd2375b525) and not an issue intrinsic to the test; and a subsequent retry makes the test pass.
- **Unskipping test.**


## Flaky test runs

- https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3728
- https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3729 [CONTROL - NO CHANGES]
@kibanamachine kibanamachine reopened this Oct 27, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.11

@jpdjere
Copy link
Contributor

jpdjere commented Oct 30, 2023 •
edited
Loading

Did some research, and it looks like the socket hang up error displayed is a Kibana wide issue, usually disguising some underlying issues.

For this specific case, it looks like the Fleet's API endpoint is unstable, fails and causes the flake.

A couple of Slack convos with related discussions:

Since in this test we are installing the security_detection_engine package via the Fleet bulk install packages endpoint, it is highly probably that the API timeouts on the serverside, and this is the error that gets displayed in CI.

@jpdjere
Copy link
Contributor

jpdjere commented Nov 3, 2023 •
edited
Loading

Proposed fixes:

FOR 8.12: since this test is being migrated to the new x-pack/test/security_solution_api_integration folder in this PR, we'll run the flaky test runner on the migrated folder to ensure that test is not reproduced in the new FTR infrastructure. (see flaky test runner link)

FOR 8.11: this is where the test failed and causes to reopen this ticket. Exclusively for 8.11, we'll change the installation of the security_detection_engine package via the current Fleet bulk install packages endpoint to use the endpoint that installs a single package, and attempt to get to a more stable test (notice that the errors reported that Kibana machine are socket hangups/timouts and are solved on retry, so they do not block any pipelines)
PR with fix for 8.11

@jpdjere jpdjere mentioned this issue Nov 3, 2023
Merged
1 task
@banderror
Copy link
Contributor

@jpdjere Thank you for the update above. It was very helpful in getting some context about the changes. I reviewed both PRs.

jpdjere added a commit that referenced this issue Nov 7, 2023
@jpdjere
[SecuritySolution] [8.11] Fix fleet_integration.ts API integration te…
4ee1728 
…st flake (#170510)

Fixes: #167056

**See
[comment](#167056 (comment))
in ticket.**
**This is only the fix for 8.11.**

## Summary

- Changes the installation of the `security_detection_engine` package
from the current Fleet bulk install packages endpoint to use the
endpoint that installs a single package, in order to get to a more
stable test
- Notice that the errors reported that Kibana machine are socket
hangups/timeouts and are solved on retry, so they do not block any
pipelines)

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
@banderror
Copy link
Contributor

Should be fixed by #169951 and #170510.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jpdjere jpdjere

Labels
8.12 candidate failed-test A test failure on a tracked branch, potentially flaky-test Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Milestone
No milestone
6 participants
@mistic @jpdjere @banderror @elasticmachine @MadameSheema @kibanamachine