Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule task telemetry pre-built/immutable/elastic rules should show detection_rules telemetry data for each list type should return mutating types such as "id", "@timestamp", etc... for list of type "detection" #164318

Closed
kibanamachine opened this issue Aug 21, 2023 · 40 comments
Closed

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules·ts - detection engine api security and spaces enabled - Group 4 Detection rule type telemetry Detection rule task telemetry pre-built/immutable/elastic rules should show detection_rules telemetry data for each list type should return mutating types such as "id", "@timestamp", etc... for list of type "detection" #164318

kibanamachine opened this issue Aug 21, 2023 · 40 comments
Assignees
@yctercero
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. triage_needed

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Aug 21, 2023 •
edited
Loading

A test failed on a tracked branch

Error: retry.try timeout: Error: expected [ [ { name: 'Security Solution Detection Rule Lists Telemetry',
      passed: false,
      time_executed_in_ms: 17,
      start_time: 1692626892959,
      end_time: 1692626892976,
      error_message: 'Cannot convert undefined or null to object' } ] ] to have a length of 2 but got 1
    at Assertion.assert (expect.js:100:11)
    at Assertion.length (expect.js:374:8)
    at detection_rules.ts:391:41
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at runAttempt (retry_for_success.ts:29:15)
    at retryForSuccess (retry_for_success.ts:68:21)
    at RetryService.try (retry.ts:31:12)
    at Context.<anonymous> (detection_rules.ts:389:9)
    at Object.apply (wrap_function.js:73:16)
    at onFailure (retry_for_success.ts:17:9)
    at retryForSuccess (retry_for_success.ts:59:13)
    at RetryService.try (retry.ts:31:12)
    at Context.<anonymous> (detection_rules.ts:389:9)
    at Object.apply (wrap_function.js:73:16)

First failure: CI Build - 8.10
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Aug 21, 2023
@botelastic botelastic bot added the needs-team Issues missing a team label label Aug 21, 2023
@kibanamachine kibanamachine added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Aug 21, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Aug 21, 2023
@MadameSheema MadameSheema added triage_needed Team:Detections and Resp Security Detection Response Team Team:Detection Rule Management Security Detection Rule Management Team labels Aug 21, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.10

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.9

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.10

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.9

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.10

@banderror banderror added Team:Detection Engine Security Solution Detection Engine Area and removed Team:Detection Rule Management Security Detection Rule Management Team labels Aug 21, 2023
@banderror banderror assigned yctercero and unassigned banderror Aug 21, 2023
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.9

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@jbudz
Copy link
Member

jbudz commented Aug 21, 2023

/skip

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.10

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.10

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.10

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.10

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.9

@e40pud e40pud mentioned this issue Aug 21, 2023
Open
@e40pud
Copy link
Contributor

e40pud commented Aug 21, 2023

Very similar test failure on 8.9 branch #164334

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.9

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@yctercero
Copy link
Contributor

@elastic/security-data-analytics could you please take a look at these tests? We believe they were added on your end?

Slack convo for context - https://elastic.slack.com/archives/C02HA9E8221/p1692628714352309

@kdelemme kdelemme mentioned this issue Oct 1, 2023
Merged
@yctercero
Copy link
Contributor

Hey @kdelemme ! Did your linked PR address this test?

@kdelemme kdelemme mentioned this issue Oct 6, 2023
Merged
@jpdjere jpdjere mentioned this issue Oct 18, 2023
Merged
@jpdjere
Copy link
Contributor

jpdjere commented Oct 19, 2023

Flakiness investigated as part of: #169262

Flaky test runner shows no flakiness for this test on 125 runs. (See GROUP 4)

Closing issue but tagging @yctercero and @elastic/security-detection-engine team to keep an eye open on this issue in case it's reopened, especially considering the issue has a long history of being reported as flaky, skipped and unskipped.

@jpdjere jpdjere closed this as completed Oct 19, 2023
jpdjere added a commit that referenced this issue Oct 31, 2023
@jpdjere
[Security Solution] Unskip tests for Detection Engine and Security So…
9eff64a 
…lution API Integration tests (#169262)

## Summary

### Changes

-
`x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules.ts`
**unskipped**
-
`x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning.ts`
**unskipped**

### Related flaky test issues

**Tests already moved to
`x-pack/test/security_solution_api_integration`**

1. #169055
2. #168775
This issue is reported for the legacy test, but is now moved to
`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/ips_text_array/text_array.ts`
3. #163511
This issue is reported for the legacy test, but is now moved to
`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/keyword_text_long/keyword.ts`

- Flaky test runner for above tests:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631

**Test still in `x-pack/test/detection_engine_api_integration`**
1. #168415
2. #164318
3. #164313
4. #145776

**Other**
1. #151636

## Flaky test runner link

Part 1 - `x-pack/test/detection_engine_api_integration` - tests not yet
migrated
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3597
Part 2 - `x-pack/test/security_solution_api_integration` - tests already
moved
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Oct 31, 2023
@jpdjere
[Security Solution] Unskip tests for Detection Engine and Security So…
8c5cb65 
…lution API Integration tests (elastic#169262)

## Summary

### Changes

-
`x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules.ts`
**unskipped**
-
`x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning.ts`
**unskipped**

### Related flaky test issues

**Tests already moved to
`x-pack/test/security_solution_api_integration`**

1. elastic#169055
2. elastic#168775
This issue is reported for the legacy test, but is now moved to
`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/ips_text_array/text_array.ts`
3. elastic#163511
This issue is reported for the legacy test, but is now moved to
`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/keyword_text_long/keyword.ts`

- Flaky test runner for above tests:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631

**Test still in `x-pack/test/detection_engine_api_integration`**
1. elastic#168415
2. elastic#164318
3. elastic#164313
4. elastic#145776

**Other**
1. elastic#151636

## Flaky test runner link

Part 1 - `x-pack/test/detection_engine_api_integration` - tests not yet
migrated
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3597
Part 2 - `x-pack/test/security_solution_api_integration` - tests already
moved
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631

(cherry picked from commit 9eff64a)
jpdjere referenced this issue Oct 31, 2023
@kibanamachine @jpdjere
[8.11] [Security Solution] Unskip tests for Detection Engine and Secu…
8a3c213 
…rity Solution API Integration tests (#169262) (#170218)

# Backport

This will backport the following commits from `main` to `8.11`:
- [[Security Solution] Unskip tests for Detection Engine and Security
Solution API Integration tests
(#169262)](#169262)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Juan Pablo
Djeredjian","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-10-31T10:36:41Z","message":"[Security
Solution] Unskip tests for Detection Engine and Security Solution API
Integration tests (#169262)\n\n## Summary\r\n\r\n###
Changes\r\n\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules.ts`\r\n**unskipped**\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning.ts`\r\n**unskipped**\r\n\r\n###
Related flaky test issues\r\n\r\n**Tests already moved
to\r\n`x-pack/test/security_solution_api_integration`**\r\n\r\n1.
https://github.com/elastic/kibana/issues/169055\r\n2.
https://github.com/elastic/kibana/issues/168775\r\nThis issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/ips_text_array/text_array.ts`\r\n3.
https://github.com/elastic/kibana/issues/163511\r\nThis issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/keyword_text_long/keyword.ts`\r\n\r\n-
Flaky test runner for above
tests:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631\r\n\r\n**Test
still in `x-pack/test/detection_engine_api_integration`**\r\n1.
https://github.com/elastic/kibana/issues/168415\r\n2.
https://github.com/elastic/kibana/issues/164318\r\n3.
https://github.com/elastic/kibana/issues/164313\r\n4.
https://github.com/elastic/kibana/issues/145776\r\n\r\n**Other**\r\n1.
https://github.com/elastic/kibana/issues/151636\r\n\r\n## Flaky test
runner link\r\n\r\nPart 1 -
`x-pack/test/detection_engine_api_integration` - tests not
yet\r\nmigrated\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3597\r\nPart
2 - `x-pack/test/security_solution_api_integration` - tests
already\r\nmoved\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631","sha":"9eff64a39120e364a4f9a6b71d9501dbb1be0999","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Detections
and Resp","Team:Detection
Engine","v8.11.0","v8.12.0"],"number":169262,"url":"https://github.com/elastic/kibana/pull/169262","mergeCommit":{"message":"[Security
Solution] Unskip tests for Detection Engine and Security Solution API
Integration tests (#169262)\n\n## Summary\r\n\r\n###
Changes\r\n\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules.ts`\r\n**unskipped**\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning.ts`\r\n**unskipped**\r\n\r\n###
Related flaky test issues\r\n\r\n**Tests already moved
to\r\n`x-pack/test/security_solution_api_integration`**\r\n\r\n1.
https://github.com/elastic/kibana/issues/169055\r\n2.
https://github.com/elastic/kibana/issues/168775\r\nThis issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/ips_text_array/text_array.ts`\r\n3.
https://github.com/elastic/kibana/issues/163511\r\nThis issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/keyword_text_long/keyword.ts`\r\n\r\n-
Flaky test runner for above
tests:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631\r\n\r\n**Test
still in `x-pack/test/detection_engine_api_integration`**\r\n1.
https://github.com/elastic/kibana/issues/168415\r\n2.
https://github.com/elastic/kibana/issues/164318\r\n3.
https://github.com/elastic/kibana/issues/164313\r\n4.
https://github.com/elastic/kibana/issues/145776\r\n\r\n**Other**\r\n1.
https://github.com/elastic/kibana/issues/151636\r\n\r\n## Flaky test
runner link\r\n\r\nPart 1 -
`x-pack/test/detection_engine_api_integration` - tests not
yet\r\nmigrated\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3597\r\nPart
2 - `x-pack/test/security_solution_api_integration` - tests
already\r\nmoved\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631","sha":"9eff64a39120e364a4f9a6b71d9501dbb1be0999"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/169262","number":169262,"mergeCommit":{"message":"[Security
Solution] Unskip tests for Detection Engine and Security Solution API
Integration tests (#169262)\n\n## Summary\r\n\r\n###
Changes\r\n\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/group4/telemetry/task_based/detection_rules.ts`\r\n**unskipped**\r\n-\r\n`x-pack/test/detection_engine_api_integration/security_and_spaces/rule_execution_logic/machine_learning.ts`\r\n**unskipped**\r\n\r\n###
Related flaky test issues\r\n\r\n**Tests already moved
to\r\n`x-pack/test/security_solution_api_integration`**\r\n\r\n1.
https://github.com/elastic/kibana/issues/169055\r\n2.
https://github.com/elastic/kibana/issues/168775\r\nThis issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/ips_text_array/text_array.ts`\r\n3.
https://github.com/elastic/kibana/issues/163511\r\nThis issue is
reported for the legacy test, but is now moved
to\r\n`x-pack/test/security_solution_api_integration/test_suites/detections_response/default_license/exceptions/operators_data_types/keyword_text_long/keyword.ts`\r\n\r\n-
Flaky test runner for above
tests:\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631\r\n\r\n**Test
still in `x-pack/test/detection_engine_api_integration`**\r\n1.
https://github.com/elastic/kibana/issues/168415\r\n2.
https://github.com/elastic/kibana/issues/164318\r\n3.
https://github.com/elastic/kibana/issues/164313\r\n4.
https://github.com/elastic/kibana/issues/145776\r\n\r\n**Other**\r\n1.
https://github.com/elastic/kibana/issues/151636\r\n\r\n## Flaky test
runner link\r\n\r\nPart 1 -
`x-pack/test/detection_engine_api_integration` - tests not
yet\r\nmigrated\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3597\r\nPart
2 - `x-pack/test/security_solution_api_integration` - tests
already\r\nmoved\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3631","sha":"9eff64a39120e364a4f9a6b71d9501dbb1be0999"}}]}]
BACKPORT-->

Co-authored-by: Juan Pablo Djeredjian <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yctercero yctercero

Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Detection Engine Security Solution Detection Engine Area Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. triage_needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@e40pud @jbudz @jpdjere @banderror @yctercero @elasticmachine @MadameSheema @kibanamachine