From ad5cdcd757475d885426d35abcc32f44594d6d60 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Tue, 27 Jun 2023 23:44:01 +0200 Subject: [PATCH 01/13] Upload MDX format --- ...rebuilt_rules_install_update_workflows.mdx | 180 ++++++++++++++++++ 1 file changed, 180 insertions(+) create mode 100644 x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx new file mode 100644 index 0000000000000..4915f4199b8b2 --- /dev/null +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx @@ -0,0 +1,180 @@ +# Rule Immutability / Customization + +## Test Plan for 2nd Milestone of Customizing prebuilt detection rules + +### Useful information + +Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.com/elastic/security-team/issues/1974) + +**Assumptions** +- The current test plan is only for Milestone 2 of the Rule Immutability/Customization feature to be released in 8.9. It does not pretend to cover any scenario for past or future milestones. Scenarios and flows are sensitive to change in future Milestones. +- Below scenarios only apply to prebuilt rules. +- Most of our users are on the 7.17.x version, that’s why the 8.x version is specified on scenarios, because this TestPlan is considering a minimum version of 8.x. +- The rule Customization feature should be available to users on the Basic license and higher. + + +### Scenarios + +### Notifications + +#### **Scenario: No callout messages are displayed when user does not have prebuilt rules installed** + +**GIVEN** user doesn't have any 8.x prebuilt rules installed +**AND** user is running a fresh instance +**WHEN** user navigates to the Rules Management Page +**THEN** no callouts message should be displayed + +#### **Scenario: No callout messages are displayed when there are no pending installs/updates** + +**GIVEN** user has the latest version of prebuilt rules `` +**WHEN** user navigates to the Rules Management Page +**THEN** no callout message is displayed for `` rules + +*CASE 1: `` = installed* +*CASE 2: `` = updated* + +#### **Scenario: Callout message is displayed when there are new prebuilt rules available to install** + +**GIVEN** user already has 8.x prebuilt rules installed +**AND** there are new prebuilt rules available to install +**WHEN** user navigates to the Rules Management Page + **THEN** user should see a callout message to install new prebuilt rules +**AND** the number of new rules available to install should be displayed on the +Add Elastic Rules link + +#### **Scenario: Callout message is displayed when there are new updates on already installed prebuilt rules** + +**GIVEN** user already has 8.x prebuilt rules installed +**AND** there are new updates available for those prebuilt rules +**WHEN** users navigate to the Rules Management Page +**THEN** users should see an update callout message +**AND** the number of outdated rules should be displayed on the Rules Updates tab + +#### **Scenario: User is notified of available prebuilt rules to install when a rule is deleted** + +**GIVEN** user has the latest version of prebuilt rules installed +**WHEN** user navigates to Rules Management Page +**AND** user deletes some prebuilt rules +**THEN** user should see a callout message with the same amount of prebuilt rules ready to install + + + + +### Prebuilt Rules Installation + +#### **Scenario: User without any installed prebuilt rule can install `` prebuilt rules** + +**GIVEN** a user that doesn’t have prebuilt rules installed +**WHEN** user navigates to Add Elastic Rules Page +**THEN** available prebuilt rules are displayed on Elastic Rules table +**AND** user can install `` prebuilt Rules +**AND** successfully installed message is displayed after installation +**AND** installed rules are removed from Elastic Rules table +**AND** rules to install counter is decreased accordingly + +*CASE 1: `` = All* +*CASE 2: `` = Selected* + +#### **Scenario: User performing a clean install for prebuilt rules sees a loading skeleton until installation is completed** + +**GIVEN** a user that is on Rules Management Page +**WHEN** user installs all prebuilt rules through Add Elastic Rules button/link +**THEN** a loading skeleton is displayed until the installation is completed + +**Prebuilt Rules Update** + +#### **Scenario: Users can update prebuilt rules** + +**GIVEN** user already has 8.x prebuilt rules installed in Kibana +**AND** there are new updates available for those prebuilt rules +**AND** user is on Rules Management Page +**WHEN** user navigates to the Rules Update tab +**THEN** user should see all the prebuilt rules +**AND** user can update outdated prebuilt rules +**AND** successfully updated message is displayed +**AND** Rules Upgrade tab counter is decreased according to the number of updated rules + + + + +### Installation / Update Failure + +#### **Scenario: Error message is displayed when any prebuilt rules operation fails** + +**GIVEN** user is `` prebuilt rules +**WHEN** the installation or update process fails +**THEN** user should see an error message +**AND** prebuilt rules are not installed/updated +**AND** the callout message for pending installs/updates is still displayed on Rules Management Page + +*CASE 1: `` = installing all* +*CASE 2: `` = installing selected* +*CASE 3: `` = Updating selected* + + + +### Add Elastic Rules Page + +#### **Scenario: New workflow elements are displayed on Rules Management Page** + +**GIVEN** a user that doesn’t have `security_detection_engine` package installed +**WHEN** user is on Rules Management Page +**THEN** “+Add Elastic rules” menu with available Rules counter is displayed +**AND** Rule Updates tab is displayed +**AND** “+Add Elastic rules” button is displayed on empty Rules Table + +#### **Scenario: Rules settings persist on Add Elastic Rules table** + +**GIVEN** a user has Rules listed on Add Elastic Rules page +**WHEN** user reloads the page +**THEN** the rule state should persist for all the rules +*CASE 1: after refreshing the table* +*CASE 2: after switching table pagination* +*CASE 3: After filtering and clear filters* + +#### **Scenario: User can navigate back to Rules Management page** + +**GIVEN** a user is on Add Rules Page +**WHEN** user navigates back to Rules Management page +**THEN** Rules Management Page is properly displayed + +#### **Scenario: User can filter prebuilt rules by query or by tag** + +**GIVEN** a user is on Add Rules Page +**WHEN** user filters by `` +**THEN** Add Rules Table is properly updated + +*CASE 1: `` = Query filter on search bar* +*CASE 2: `` = Tag filter* + + + + + +### Authorization / RBAC + +#### **Scenario: User with read privileges on security solution cannot install prebuilt rules** + +**GIVEN** a user with Security: read privileges on Security solution +**WHEN** user navigates to Add Elastic Rules Page +**THEN** user can see available prebuilt rules to install +**AND** user cannot Install those prebuilt rules + +#### **Scenario: User with read privileges on security solution cannot update prebuilt rules** + +**GIVEN** a user with Security: read privileges on Security solution +**WHEN** user navigates to Rule Updates Tab on Rules Management Page +**THEN** user can see new updates for installed prebuilt rules +**AND** user cannot Update those prebuilt rules + +**Kibana upgrade** + +#### **Scenario: User can operate with prebuilt rules when user upgrades from version `` to 8.9 version** + +**GIVEN** a user that is upgrading from version `` to version 8.9 +**AND** the `` instance contains already installed prebuilt rules +**WHEN** the upgrade is complete +**THEN** user can install new prebuilt rules +**AND** remove installed prebuilt rules +**AND** update prebuilt rules from `` to 8.9 + +*version: 8.7, 7.17.x* \ No newline at end of file From ab577e188355859b7af2a4a4ff95f50a726df822 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Wed, 28 Jun 2023 00:06:43 +0200 Subject: [PATCH 02/13] Line fix --- .../prebuilt_rules_install_update_workflows.mdx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx index 4915f4199b8b2..53945376f2a5a 100644 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx @@ -80,7 +80,11 @@ Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.co **WHEN** user installs all prebuilt rules through Add Elastic Rules button/link **THEN** a loading skeleton is displayed until the installation is completed -**Prebuilt Rules Update** + + + + +### Prebuilt Rules Update #### **Scenario: Users can update prebuilt rules** From bd24518c96eeef9d85d9e4a4d9a6e49302a237c3 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Wed, 28 Jun 2023 00:23:25 +0200 Subject: [PATCH 03/13] Fixes --- .../prebuilt_rules_install_update_workflows.mdx | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx index 53945376f2a5a..4015337f63752 100644 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx @@ -170,7 +170,10 @@ Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.co **THEN** user can see new updates for installed prebuilt rules **AND** user cannot Update those prebuilt rules -**Kibana upgrade** + + + +### Kibana upgrade #### **Scenario: User can operate with prebuilt rules when user upgrades from version `` to 8.9 version** @@ -181,4 +184,7 @@ Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.co **AND** remove installed prebuilt rules **AND** update prebuilt rules from `` to 8.9 -*version: 8.7, 7.17.x* \ No newline at end of file +| version | +|----------| +| 8.7 | +| 7.17.x | \ No newline at end of file From 1308860e410aa26a278870e2871a2055aa95519b Mon Sep 17 00:00:00 2001 From: jpdjere Date: Wed, 28 Jun 2023 13:54:39 +0200 Subject: [PATCH 04/13] Changes after syncing with @vgomez-el --- ...rebuilt_rules_install_update_workflows.mdx | 91 ++++++++++++++----- 1 file changed, 70 insertions(+), 21 deletions(-) diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx index 4015337f63752..eb618df50f93e 100644 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx @@ -1,10 +1,10 @@ -# Rule Immutability / Customization +# Prebuilt Rules Install and Update workflows ## Test Plan for 2nd Milestone of Customizing prebuilt detection rules ### Useful information -Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.com/elastic/security-team/issues/1974) +Part of epic: [https://github.com/elastic/security-team/issues/1974](https://github.com/elastic/security-team/issues/1974) **Assumptions** - The current test plan is only for Milestone 2 of the Rule Immutability/Customization feature to be released in 8.9. It does not pretend to cover any scenario for past or future milestones. Scenarios and flows are sensitive to change in future Milestones. @@ -17,18 +17,12 @@ Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.co ### Notifications -#### **Scenario: No callout messages are displayed when user does not have prebuilt rules installed** - -**GIVEN** user doesn't have any 8.x prebuilt rules installed -**AND** user is running a fresh instance -**WHEN** user navigates to the Rules Management Page -**THEN** no callouts message should be displayed - #### **Scenario: No callout messages are displayed when there are no pending installs/updates** **GIVEN** user has the latest version of prebuilt rules `` **WHEN** user navigates to the Rules Management Page **THEN** no callout message is displayed for `` rules +**AND** no badges with number of available rules to install/update are displayed *CASE 1: `` = installed* *CASE 2: `` = updated* @@ -55,8 +49,22 @@ Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.co **WHEN** user navigates to Rules Management Page **AND** user deletes some prebuilt rules **THEN** user should see a callout message with the same amount of prebuilt rules ready to install +**AND** the updated number of available rules to install should be displayed on the +Add Elastic Rules link + +#### **Scenario: User is notified that all available rules have been installed in the Add Elastic Rules** + +**GIVEN** user has all available rules installed +**WHEN** user navigates to the Add Elastic Rules Page +**THEN** user should see a message indicating that all available rules have been installed +**AND** user should see a CTA that leads to the Rules Management Page +**AND** user should navigate back to Rules Management Page when clicking on the CTA +#### **Scenario: User is notified that all installed rules are up to date in the Rule Updates tab** + +**GIVEN** user has all available rules installed +**WHEN** user navigates to the Rule Update +**THEN** user should see a message indicating that all installed rules are up to date ### Prebuilt Rules Installation @@ -73,11 +81,12 @@ Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.co *CASE 1: `` = All* *CASE 2: `` = Selected* +*CASE 3: `` = Individual* -#### **Scenario: User performing a clean install for prebuilt rules sees a loading skeleton until installation is completed** +#### **Scenario: User navigating to the Add Elastic Rules page sees a loading skeleton until the prebuilt rules package installation is completed** **GIVEN** a user that is on Rules Management Page -**WHEN** user installs all prebuilt rules through Add Elastic Rules button/link +**WHEN** user to the Add Elastic Rules page before rules package is installed **THEN** a loading skeleton is displayed until the installation is completed @@ -92,7 +101,7 @@ Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.co **AND** there are new updates available for those prebuilt rules **AND** user is on Rules Management Page **WHEN** user navigates to the Rules Update tab -**THEN** user should see all the prebuilt rules +**THEN** user should see all the prebuilt rules that have updates available **AND** user can update outdated prebuilt rules **AND** successfully updated message is displayed **AND** Rules Upgrade tab counter is decreased according to the number of updated rules @@ -100,7 +109,7 @@ Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.co -### Installation / Update Failure +### Package Installation / Rule Installation / Rule Update Failure #### **Scenario: Error message is displayed when any prebuilt rules operation fails** @@ -109,10 +118,22 @@ Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.co **THEN** user should see an error message **AND** prebuilt rules are not installed/updated **AND** the callout message for pending installs/updates is still displayed on Rules Management Page +**AND** the number of available rules to install and upgrade in the badges does not change. + +*CASE 1: `` = installing all* +*CASE 2: `` = installing selected* +*CASE 3: `` = installing individual* +*CASE 4: `` = Updating all* +*CASE 5: `` = Updating selected* +*CASE 6: `` = Updating individual* -*CASE 1: `` = installing all* -*CASE 2: `` = installing selected* -*CASE 3: `` = Updating selected* +#### **Scenario: No callout messages are displayed when rule package isntallation fails and no rules are avialble for install/update** + +**GIVEN** user navigates to Rules Management Page +**AND** user is running a fresh instance +**AND** rule package installation fails +**THEN** no callouts message should be displayed +**AND** the number of available rules to install and upgrade in the badges does not change. @@ -129,9 +150,10 @@ Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.co #### **Scenario: Rules settings persist on Add Elastic Rules table** **GIVEN** a user has Rules listed on Add Elastic Rules page -**WHEN** user reloads the page -**THEN** the rule state should persist for all the rules -*CASE 1: after refreshing the table* +**WHEN** +**THEN** the available rules state should persist for all the rules + +*CASE 1: user reloads the page* *CASE 2: after switching table pagination* *CASE 3: After filtering and clear filters* @@ -141,17 +163,44 @@ Ticket: [https://github.com/elastic/security-team/issues/1974](https://github.co **WHEN** user navigates back to Rules Management page **THEN** Rules Management Page is properly displayed -#### **Scenario: User can filter prebuilt rules by query or by tag** +#### **Scenario: User can filter prebuilt rules by rule name or by tag** **GIVEN** a user is on Add Rules Page **WHEN** user filters by `` **THEN** Add Rules Table is properly updated -*CASE 1: `` = Query filter on search bar* +*CASE 1: `` = rule name on search bar* *CASE 2: `` = Tag filter* +### Rule Updates tab + +#### **Scenario: Rules settings persist on Rule Updates table** + +**GIVEN** a user has Rules listed on Rule Updates table +**WHEN** +**THEN** the rules with available updates state should persist + +*CASE 1: user reloads the page* +*CASE 2: after switching table pagination* +*CASE 3: After filtering and clear filters* + +#### **Scenario: User can navigate back to Rules Management tab** + +**GIVEN** a user is on Rule Updates tab +**WHEN** user navigates back to Rules Management page +**THEN** Rules Management Page is properly displayed + +#### **Scenario: User can filter prebuilt rules by rule name or by tag** + +**GIVEN** a user is on Rule Updates tab +**WHEN** user filters by `` +**THEN** Rule Updates tab is properly updated + +*CASE 1: `` = rule name on search bar* +*CASE 2: `` = Tag filter* + ### Authorization / RBAC From 4fc6fb8d3347c64be031c9898408d60d77f0aaf4 Mon Sep 17 00:00:00 2001 From: jpdjere Date: Wed, 28 Jun 2023 14:00:26 +0200 Subject: [PATCH 05/13] fixes --- ...rebuilt_rules_install_update_workflows.mdx | 56 +++++++++---------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx index eb618df50f93e..cf258738cd2a4 100644 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx @@ -24,8 +24,8 @@ Part of epic: [https://github.com/elastic/security-team/issues/1974](https://git **THEN** no callout message is displayed for `` rules **AND** no badges with number of available rules to install/update are displayed -*CASE 1: `` = installed* -*CASE 2: `` = updated* +*CASE 1: `` = to install* +*CASE 2: `` = to update* #### **Scenario: Callout message is displayed when there are new prebuilt rules available to install** @@ -120,20 +120,20 @@ Part of epic: [https://github.com/elastic/security-team/issues/1974](https://git **AND** the callout message for pending installs/updates is still displayed on Rules Management Page **AND** the number of available rules to install and upgrade in the badges does not change. -*CASE 1: `` = installing all* -*CASE 2: `` = installing selected* -*CASE 3: `` = installing individual* -*CASE 4: `` = Updating all* -*CASE 5: `` = Updating selected* -*CASE 6: `` = Updating individual* +*CASE 1: `` = installing all* +*CASE 2: `` = installing selected* +*CASE 3: `` = installing individual* +*CASE 4: `` = Updating all* +*CASE 5: `` = Updating selected* +*CASE 6: `` = Updating individual* #### **Scenario: No callout messages are displayed when rule package isntallation fails and no rules are avialble for install/update** -**GIVEN** user navigates to Rules Management Page +**GIVEN** user navigates to Rules Management Page **AND** user is running a fresh instance -**AND** rule package installation fails -**THEN** no callouts message should be displayed -**AND** the number of available rules to install and upgrade in the badges does not change. +**AND** rule package installation fails +**THEN** no callouts message should be displayed +**AND** the number of available rules to install and upgrade in the badges does not change. @@ -143,14 +143,14 @@ Part of epic: [https://github.com/elastic/security-team/issues/1974](https://git **GIVEN** a user that doesn’t have `security_detection_engine` package installed **WHEN** user is on Rules Management Page -**THEN** “+Add Elastic rules” menu with available Rules counter is displayed -**AND** Rule Updates tab is displayed -**AND** “+Add Elastic rules” button is displayed on empty Rules Table +**THEN** “+Add Elastic rules” menu with available Rules counter is displayed +**AND** Rule Updates tab is displayed +**AND** “+Add Elastic rules” button is displayed on empty Rules Table #### **Scenario: Rules settings persist on Add Elastic Rules table** **GIVEN** a user has Rules listed on Add Elastic Rules page -**WHEN** +**WHEN** `` **THEN** the available rules state should persist for all the rules *CASE 1: user reloads the page* @@ -179,18 +179,18 @@ Part of epic: [https://github.com/elastic/security-team/issues/1974](https://git #### **Scenario: Rules settings persist on Rule Updates table** **GIVEN** a user has Rules listed on Rule Updates table -**WHEN** +**WHEN** `` **THEN** the rules with available updates state should persist -*CASE 1: user reloads the page* -*CASE 2: after switching table pagination* -*CASE 3: After filtering and clear filters* +*CASE 1: user reloads the page* +*CASE 2: after switching table pagination* +*CASE 3: After filtering and clear filters* #### **Scenario: User can navigate back to Rules Management tab** -**GIVEN** a user is on Rule Updates tab -**WHEN** user navigates back to Rules Management page -**THEN** Rules Management Page is properly displayed +**GIVEN** a user is on Rule Updates tab +**WHEN** user navigates back to Rules Management page +**THEN** Rules Management Page is properly displayed #### **Scenario: User can filter prebuilt rules by rule name or by tag** @@ -209,14 +209,14 @@ Part of epic: [https://github.com/elastic/security-team/issues/1974](https://git **GIVEN** a user with Security: read privileges on Security solution **WHEN** user navigates to Add Elastic Rules Page -**THEN** user can see available prebuilt rules to install +**THEN** user can see available prebuilt rules to install **AND** user cannot Install those prebuilt rules #### **Scenario: User with read privileges on security solution cannot update prebuilt rules** **GIVEN** a user with Security: read privileges on Security solution **WHEN** user navigates to Rule Updates Tab on Rules Management Page -**THEN** user can see new updates for installed prebuilt rules +**THEN** user can see new updates for installed prebuilt rules **AND** user cannot Update those prebuilt rules @@ -229,9 +229,9 @@ Part of epic: [https://github.com/elastic/security-team/issues/1974](https://git **GIVEN** a user that is upgrading from version `` to version 8.9 **AND** the `` instance contains already installed prebuilt rules **WHEN** the upgrade is complete -**THEN** user can install new prebuilt rules -**AND** remove installed prebuilt rules -**AND** update prebuilt rules from `` to 8.9 +**THEN** user can install new prebuilt rules +**AND** remove installed prebuilt rules +**AND** update prebuilt rules from `` to 8.9 | version | |----------| From fce12807841d98b0c58a04c7e5d0053c90c59906 Mon Sep 17 00:00:00 2001 From: Georgii Gorbachev Date: Fri, 30 Jun 2023 16:46:50 +0200 Subject: [PATCH 06/13] Change the format of the test plan (Gherkin syntax, etc) --- .../installation_and_upgrade.md | 285 ++++++++++++++++++ ...rebuilt_rules_install_update_workflows.mdx | 239 --------------- .../detection_response/prebuilt_rules/todo | 1 - 3 files changed, 285 insertions(+), 240 deletions(-) create mode 100644 x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md delete mode 100644 x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx delete mode 100644 x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/todo diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md new file mode 100644 index 0000000000000..4d77e8a86a9ce --- /dev/null +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md @@ -0,0 +1,285 @@ +# Installation and Upgrade of Prebuilt Rules + +This is a test plan for the workflows of installing and upgrading prebuilt rules. + +Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic. It does not cover any past functionality that was removed or any functionality to be implemented in the future. The plan is about to change in the future Milestones. + +## Useful information + +### Related tickets + +- [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic +- [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176) +- [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192) +- [Document the new UI for installing and upgrading prebuilt detection rules](https://github.com/elastic/security-docs/issues/3496) + +### Assumptions + +- Below scenarios only apply to prebuilt detection rules. +- Most of our users are on the 7.17.x version, that’s why the 8.x version is specified on scenarios, because this TestPlan is considering a minimum version of 8.x. +- Users should be able to install and upgrade prebuilt rules on the `Basic` license and higher. +- EPR (Elastic Package Registry) is available for fetching the `security_detection_engine` package unless explicitly indicated otherwise. +- Only the latest stable `security_detection_engine` package is checked for update/installation and pre-release packages are ignored. + +## Scenarios + +### Notifications + +#### **Scenario: No callout messages are displayed when there are no pending installs/updates** + +```Gherkin +Given user has the latest version of prebuilt rules +When user navigates to the Rule Management page +Then no callout message is displayed for rules +And no badges with number of available rules to install/update are displayed + +Examples: + | prebuilt_rules_status | + | to install | + | to update | +``` + +#### **Scenario: Callout message is displayed when there are new prebuilt rules available to install** + +```Gherkin +Given user already has 8.x prebuilt rules installed +And there are new prebuilt rules available to install +When user navigates to the Rules Management Page +Then user should see a callout message to install new prebuilt rules +And the number of new rules available to install should be displayed on the +Add Elastic Rules link +``` + +#### **Scenario: Callout message is displayed when there are new updates on already installed prebuilt rules** + +```Gherkin +Given user already has 8.x prebuilt rules installed +And there are new updates available for those prebuilt rules +When users navigate to the Rules Management Page +Then users should see an update callout message +And the number of outdated rules should be displayed on the Rules Updates tab +``` + +#### **Scenario: User is notified of available prebuilt rules to install when a rule is deleted** + +```Gherkin +Given user has the latest version of prebuilt rules installed +When user navigates to Rules Management Page +And user deletes some prebuilt rules +Then user should see a callout message with the same amount of prebuilt rules ready to install +And the updated number of available rules to install should be displayed on the +Add Elastic Rules link +``` + +#### **Scenario: User is notified that all available rules have been installed in the Add Elastic Rules** + +```Gherkin +Given user has all available rules installed +When user navigates to the Add Elastic Rules Page +Then user should see a message indicating that all available rules have been installed +And user should see a CTA that leads to the Rules Management Page +And user should navigate back to Rules Management Page when clicking on the CTA +``` + +#### **Scenario: User is notified that all installed rules are up to date in the Rule Updates tab** + +```Gherkin +Given user has all available rules installed +When user navigates to the Rule Update +Then user should see a message indicating that all installed rules are up to date +``` + +### Prebuilt Rules Installation + +#### **Scenario: User without any installed prebuilt rules can install `` prebuilt rules** + +```Gherkin +Given a user that doesn’t have prebuilt rules installed +When user navigates to Add Elastic Rules Page +Then available prebuilt rules are displayed on Elastic Rules table +And user can install prebuilt Rules +And successfully installed message is displayed after installation +And installed rules are removed from Elastic Rules table +And rules to install counter is decreased accordingly + +Examples: + | amount | + | all | + | selected | + | individual | +``` + +#### **Scenario: User navigating to the Add Elastic Rules page sees a loading skeleton until the prebuilt rules package installation is completed** + +```Gherkin +Given a user that is on Rules Management Page +When user to the Add Elastic Rules page before rules package is installed +Then a loading skeleton is displayed until the installation is completed +``` + +### Prebuilt Rules Update + +#### **Scenario: Users can update prebuilt rules** + +```Gherkin +Given user already has 8.x prebuilt rules installed in Kibana +And there are new updates available for those prebuilt rules +And user is on Rules Management Page +When user navigates to the Rules Update tab +Then user should see all the prebuilt rules that have updates available +And user can update outdated prebuilt rules +And successfully updated message is displayed +And Rules Upgrade tab counter is decreased according to the number of updated rules +``` + +### Package Installation / Rule Installation / Rule Update Failure + +#### **Scenario: Error message is displayed when any prebuilt rules operation fails** + +```Gherkin +Given user is prebuilt rules +When the installation or update process fails +Then user should see an error message +And prebuilt rules are not installed/updated +And the callout message for pending installs/updates is still displayed on Rules Management Page +And the number of available rules to install and upgrade in the badges does not change + +Examples: + | action | + | installing all | + | installing selected | + | installing individual | + | updating all | + | updating selected | + | updating individual | +``` + +#### **Scenario: No callout messages are displayed when rule package installation fails and no rules are avialble for install/update** + +```Gherkin +Given user navigates to Rules Management Page +And user is running a fresh instance +And rule package installation fails +Then no callouts message should be displayed +And the number of available rules to install and upgrade in the badges does not change +``` + +### Add Elastic Rules Page + +#### **Scenario: New workflow elements are displayed on Rules Management Page** + +```Gherkin +Given a user that doesn’t have `security_detection_engine` package installed +When user is on Rules Management Page +Then "+Add Elastic rules" menu with available Rules counter is displayed +And Rule Updates tab is displayed +And "+Add Elastic rules" button is displayed on empty Rules Table +``` + +#### **Scenario: Rules settings persist on Add Elastic Rules table** + +```Gherkin +Given a user has Rules listed on Add Elastic Rules page +When +Then the available rules state should persist for all the rules + +Examples: + | case | + | user reloads the page | + | after switching table pagination | + | after filtering and clear filters | +``` + +#### **Scenario: User can navigate back to Rules Management page** + +```Gherkin +Given a user is on Add Rules Page +When user navigates back to Rules Management page +Then Rules Management Page is properly displayed +``` + +#### **Scenario: User can filter prebuilt rules by rule name or by tag** + +```Gherkin +Given a user is on Add Rules Page +When user filters by +Then Add Rules Table is properly updated + +Examples: + | filter | + | rule name on search bar | + | Tag filter | +``` + +### Rule Updates tab + +#### **Scenario: Rules settings persist on Rule Updates table** + +```Gherkin +Given a user has Rules listed on Rule Updates table +When +Then the rules with available updates state should persist + +Examples: + | case | + | user reloads the page | + | after switching table pagination | + | after filtering and clear filters | +``` + +#### **Scenario: User can navigate back to Rules Management tab** + +```Gherkin +Given a user is on Rule Updates tab +When user navigates back to Rules Management page +Then Rules Management Page is properly displayed +``` + +#### **Scenario: User can filter prebuilt rules by rule name or by tag** + +```Gherkin +Given a user is on Rule Updates tab +When user filters by +Then Rule Updates tab is properly updated + +Examples: + | filter | + | rule name on search bar | + | Tag filter | +``` + +### Authorization / RBAC + +#### **Scenario: User with read privileges on security solution cannot install prebuilt rules** + +```Gherkin +Given a user with Security: read privileges on Security solution +When user navigates to Add Elastic Rules Page +Then user can see available prebuilt rules to install +And user cannot Install those prebuilt rules +``` + +#### **Scenario: User with read privileges on security solution cannot update prebuilt rules** + +```Gherkin +Given a user with Security: read privileges on Security solution +When user navigates to Rule Updates Tab on Rules Management Page +Then user can see new updates for installed prebuilt rules +And user cannot Update those prebuilt rules +``` + +### Kibana upgrade + +#### **Scenario: User can operate with prebuilt rules when user upgrades from version `` to 8.9 version** + +```Gherkin +Given a user that is upgrading from version to version 8.9 +And the instance contains already installed prebuilt rules +When the upgrade is complete +Then user can install new prebuilt rules +And remove installed prebuilt rules +And update prebuilt rules from to 8.9 + +Examples: + | version | + | 8.7 | + | 7.17.x | +``` diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx deleted file mode 100644 index cf258738cd2a4..0000000000000 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/prebuilt_rules_install_update_workflows.mdx +++ /dev/null @@ -1,239 +0,0 @@ -# Prebuilt Rules Install and Update workflows - -## Test Plan for 2nd Milestone of Customizing prebuilt detection rules - -### Useful information - -Part of epic: [https://github.com/elastic/security-team/issues/1974](https://github.com/elastic/security-team/issues/1974) - -**Assumptions** -- The current test plan is only for Milestone 2 of the Rule Immutability/Customization feature to be released in 8.9. It does not pretend to cover any scenario for past or future milestones. Scenarios and flows are sensitive to change in future Milestones. -- Below scenarios only apply to prebuilt rules. -- Most of our users are on the 7.17.x version, that’s why the 8.x version is specified on scenarios, because this TestPlan is considering a minimum version of 8.x. -- The rule Customization feature should be available to users on the Basic license and higher. - - -### Scenarios - -### Notifications - -#### **Scenario: No callout messages are displayed when there are no pending installs/updates** - -**GIVEN** user has the latest version of prebuilt rules `` -**WHEN** user navigates to the Rules Management Page -**THEN** no callout message is displayed for `` rules -**AND** no badges with number of available rules to install/update are displayed - -*CASE 1: `` = to install* -*CASE 2: `` = to update* - -#### **Scenario: Callout message is displayed when there are new prebuilt rules available to install** - -**GIVEN** user already has 8.x prebuilt rules installed -**AND** there are new prebuilt rules available to install -**WHEN** user navigates to the Rules Management Page - **THEN** user should see a callout message to install new prebuilt rules -**AND** the number of new rules available to install should be displayed on the +Add Elastic Rules link - -#### **Scenario: Callout message is displayed when there are new updates on already installed prebuilt rules** - -**GIVEN** user already has 8.x prebuilt rules installed -**AND** there are new updates available for those prebuilt rules -**WHEN** users navigate to the Rules Management Page -**THEN** users should see an update callout message -**AND** the number of outdated rules should be displayed on the Rules Updates tab - -#### **Scenario: User is notified of available prebuilt rules to install when a rule is deleted** - -**GIVEN** user has the latest version of prebuilt rules installed -**WHEN** user navigates to Rules Management Page -**AND** user deletes some prebuilt rules -**THEN** user should see a callout message with the same amount of prebuilt rules ready to install -**AND** the updated number of available rules to install should be displayed on the +Add Elastic Rules link - -#### **Scenario: User is notified that all available rules have been installed in the Add Elastic Rules** - -**GIVEN** user has all available rules installed -**WHEN** user navigates to the Add Elastic Rules Page -**THEN** user should see a message indicating that all available rules have been installed -**AND** user should see a CTA that leads to the Rules Management Page -**AND** user should navigate back to Rules Management Page when clicking on the CTA - - -#### **Scenario: User is notified that all installed rules are up to date in the Rule Updates tab** - -**GIVEN** user has all available rules installed -**WHEN** user navigates to the Rule Update -**THEN** user should see a message indicating that all installed rules are up to date - - -### Prebuilt Rules Installation - -#### **Scenario: User without any installed prebuilt rule can install `` prebuilt rules** - -**GIVEN** a user that doesn’t have prebuilt rules installed -**WHEN** user navigates to Add Elastic Rules Page -**THEN** available prebuilt rules are displayed on Elastic Rules table -**AND** user can install `` prebuilt Rules -**AND** successfully installed message is displayed after installation -**AND** installed rules are removed from Elastic Rules table -**AND** rules to install counter is decreased accordingly - -*CASE 1: `` = All* -*CASE 2: `` = Selected* -*CASE 3: `` = Individual* - -#### **Scenario: User navigating to the Add Elastic Rules page sees a loading skeleton until the prebuilt rules package installation is completed** - -**GIVEN** a user that is on Rules Management Page -**WHEN** user to the Add Elastic Rules page before rules package is installed -**THEN** a loading skeleton is displayed until the installation is completed - - - - - -### Prebuilt Rules Update - -#### **Scenario: Users can update prebuilt rules** - -**GIVEN** user already has 8.x prebuilt rules installed in Kibana -**AND** there are new updates available for those prebuilt rules -**AND** user is on Rules Management Page -**WHEN** user navigates to the Rules Update tab -**THEN** user should see all the prebuilt rules that have updates available -**AND** user can update outdated prebuilt rules -**AND** successfully updated message is displayed -**AND** Rules Upgrade tab counter is decreased according to the number of updated rules - - - - -### Package Installation / Rule Installation / Rule Update Failure - -#### **Scenario: Error message is displayed when any prebuilt rules operation fails** - -**GIVEN** user is `` prebuilt rules -**WHEN** the installation or update process fails -**THEN** user should see an error message -**AND** prebuilt rules are not installed/updated -**AND** the callout message for pending installs/updates is still displayed on Rules Management Page -**AND** the number of available rules to install and upgrade in the badges does not change. - -*CASE 1: `` = installing all* -*CASE 2: `` = installing selected* -*CASE 3: `` = installing individual* -*CASE 4: `` = Updating all* -*CASE 5: `` = Updating selected* -*CASE 6: `` = Updating individual* - -#### **Scenario: No callout messages are displayed when rule package isntallation fails and no rules are avialble for install/update** - -**GIVEN** user navigates to Rules Management Page -**AND** user is running a fresh instance -**AND** rule package installation fails -**THEN** no callouts message should be displayed -**AND** the number of available rules to install and upgrade in the badges does not change. - - - -### Add Elastic Rules Page - -#### **Scenario: New workflow elements are displayed on Rules Management Page** - -**GIVEN** a user that doesn’t have `security_detection_engine` package installed -**WHEN** user is on Rules Management Page -**THEN** “+Add Elastic rules” menu with available Rules counter is displayed -**AND** Rule Updates tab is displayed -**AND** “+Add Elastic rules” button is displayed on empty Rules Table - -#### **Scenario: Rules settings persist on Add Elastic Rules table** - -**GIVEN** a user has Rules listed on Add Elastic Rules page -**WHEN** `` -**THEN** the available rules state should persist for all the rules - -*CASE 1: user reloads the page* -*CASE 2: after switching table pagination* -*CASE 3: After filtering and clear filters* - -#### **Scenario: User can navigate back to Rules Management page** - -**GIVEN** a user is on Add Rules Page -**WHEN** user navigates back to Rules Management page -**THEN** Rules Management Page is properly displayed - -#### **Scenario: User can filter prebuilt rules by rule name or by tag** - -**GIVEN** a user is on Add Rules Page -**WHEN** user filters by `` -**THEN** Add Rules Table is properly updated - -*CASE 1: `` = rule name on search bar* -*CASE 2: `` = Tag filter* - - - -### Rule Updates tab - -#### **Scenario: Rules settings persist on Rule Updates table** - -**GIVEN** a user has Rules listed on Rule Updates table -**WHEN** `` -**THEN** the rules with available updates state should persist - -*CASE 1: user reloads the page* -*CASE 2: after switching table pagination* -*CASE 3: After filtering and clear filters* - -#### **Scenario: User can navigate back to Rules Management tab** - -**GIVEN** a user is on Rule Updates tab -**WHEN** user navigates back to Rules Management page -**THEN** Rules Management Page is properly displayed - -#### **Scenario: User can filter prebuilt rules by rule name or by tag** - -**GIVEN** a user is on Rule Updates tab -**WHEN** user filters by `` -**THEN** Rule Updates tab is properly updated - -*CASE 1: `` = rule name on search bar* -*CASE 2: `` = Tag filter* - - - -### Authorization / RBAC - -#### **Scenario: User with read privileges on security solution cannot install prebuilt rules** - -**GIVEN** a user with Security: read privileges on Security solution -**WHEN** user navigates to Add Elastic Rules Page -**THEN** user can see available prebuilt rules to install -**AND** user cannot Install those prebuilt rules - -#### **Scenario: User with read privileges on security solution cannot update prebuilt rules** - -**GIVEN** a user with Security: read privileges on Security solution -**WHEN** user navigates to Rule Updates Tab on Rules Management Page -**THEN** user can see new updates for installed prebuilt rules -**AND** user cannot Update those prebuilt rules - - - - -### Kibana upgrade - -#### **Scenario: User can operate with prebuilt rules when user upgrades from version `` to 8.9 version** - -**GIVEN** a user that is upgrading from version `` to version 8.9 -**AND** the `` instance contains already installed prebuilt rules -**WHEN** the upgrade is complete -**THEN** user can install new prebuilt rules -**AND** remove installed prebuilt rules -**AND** update prebuilt rules from `` to 8.9 - -| version | -|----------| -| 8.7 | -| 7.17.x | \ No newline at end of file diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/todo b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/todo deleted file mode 100644 index 99ffc1061d97c..0000000000000 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/todo +++ /dev/null @@ -1 +0,0 @@ -Add test plans. From 8c6e8175ddb8175129b11f5cb072c5c616f000d1 Mon Sep 17 00:00:00 2001 From: Georgii Gorbachev Date: Fri, 30 Jun 2023 21:13:30 +0200 Subject: [PATCH 07/13] Add some scenarios from the test plan for the legacy workflows --- .../installation_and_upgrade.md | 68 +++++++++++++++++-- 1 file changed, 63 insertions(+), 5 deletions(-) diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md index 4d77e8a86a9ce..de2fbd4de5866 100644 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md @@ -2,27 +2,85 @@ This is a test plan for the workflows of installing and upgrading prebuilt rules. -Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic. It does not cover any past functionality that was removed or any functionality to be implemented in the future. The plan is about to change in the future Milestones. +Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic. It does not cover any past functionality that was removed or functionality to be implemented in the future. The plan is about to change in the future Milestones. ## Useful information -### Related tickets +### Tickets - [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic - [Ensure full test coverage for existing workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148176) - [Write test plan and add test coverage for the new workflows of installing and upgrading prebuilt rules](https://github.com/elastic/kibana/issues/148192) - [Document the new UI for installing and upgrading prebuilt detection rules](https://github.com/elastic/security-docs/issues/3496) +### Terminology + +- **EPR**: [Elastic Package Registry](https://github.com/elastic/package-registry), service that hosts our **Package**. + +- **Package**: `security_detection_engine` Fleet package that we use to distribute prebuilt detection rules in the form of `security-rule` assets (saved objects). + +- **Real package**: actual latest stable package distributed and pulled from EPR via Fleet. + +- **Mock rules**: `security-rule` assets that are indexed into the `.kibana_security_solution` index directly in the test setup, either by using the ES client _in integration tests_ or by an API request _in Cypress tests_. + +- **Air-gapped environment**: an environment where Kibana doesn't have access to the internet. In general, EPR is not available in such environments, except the cases when the user runs a custom EPR inside the environment. + ### Assumptions - Below scenarios only apply to prebuilt detection rules. -- Most of our users are on the 7.17.x version, that’s why the 8.x version is specified on scenarios, because this TestPlan is considering a minimum version of 8.x. +- Most of our users are on the 7.17.x version, that’s why the 8.x version is specified on scenarios, because this test plan is considering a minimum version of 8.x. - Users should be able to install and upgrade prebuilt rules on the `Basic` license and higher. -- EPR (Elastic Package Registry) is available for fetching the `security_detection_engine` package unless explicitly indicated otherwise. -- Only the latest stable `security_detection_engine` package is checked for update/installation and pre-release packages are ignored. +- EPR is available for fetching the package unless explicitly indicated otherwise. +- Only the latest **stable** package is checked for installation/upgrade and pre-release packages are ignored. + +### Non-functional requirements + +- Notifications, rule installation and rule upgrade workflows should work: + - regardless of the package type: with historical rule versions or without; + - regardless of the package registry availability: i.e., they should also work in air-gapped environments. +- Rule installation and upgrade workflows should work with packages containing up to 15000 historical rule versions. This is the max number of versions of all rules in the package. This limit is enforced by Fleet. +- Kibana should not crash with Out Of Memory exception during package installation. +- For test purposes, it should be possible to use detection rules package versions lower than the latest. ## Scenarios +### Prebuilt rules package installation + +#### **Scenario: Package is installed via Fleet** + +**Coverage**: 0 tests - covered by other scenarios. This scenario will be covered by the **Users install the latest prebuilt rules** scenario, which will include 1 e2e test which installs the rules from the real package. + +```Gherkin +Given user doesn't have the package installed +When they navigate to the Rule Management page +Then the package is installed in the background from EPR +``` + +#### **Scenario: Package is installed via bundled Fleet package in Kibana** + +**Coverage**: 1 integration test. + +```Gherkin +Given user doesn't have the package installed +And the user is in an air-gapped environment +When they navigate to the Rule Management page +Then the package is installed in the background from packages bundled into Kibana +``` + +#### **Scenario: Large package can be installed on a small Kibana instance** + +**Coverage**: 1 integration test. + +```Gherkin +Given user doesn't have the package installed +And the package has the largest amount of historical rule versions installed (15000) +And the Kibana instance has a memory heap size of X Mb (see note below) +When they navigate to the Rule Management page +Then the package is installed without Kibana crashing with an Out Of Memory error +``` + +**Note**: The amount of memory is undefined as of now because, during implementation, we will try to find a memory heap threshold below which Kibana starts to crash constantly when you install the package with 15k rules. The plan is to then increase it to the point where it stops crashing, and use it as our value for this test. + ### Notifications #### **Scenario: No callout messages are displayed when there are no pending installs/updates** From 62fce6b8628b6b23b0a8e50f1ea4a0962979f0c5 Mon Sep 17 00:00:00 2001 From: Georgii Gorbachev Date: Sat, 1 Jul 2023 21:53:47 +0200 Subject: [PATCH 08/13] Add scenarios for the API, edit more scenarios --- .../installation_and_upgrade.md | 367 ++++++++++++++---- .../prebuilt_rules/original_scenarios.md | 264 +++++++++++++ 2 files changed, 560 insertions(+), 71 deletions(-) create mode 100644 x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/original_scenarios.md diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md index de2fbd4de5866..1d34fa29c83a9 100644 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md @@ -25,6 +25,10 @@ Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule - **Air-gapped environment**: an environment where Kibana doesn't have access to the internet. In general, EPR is not available in such environments, except the cases when the user runs a custom EPR inside the environment. +- **CTA**: "call to action", usually a button, a link, or a callout message with a button, etc, that invites the user to do some action. + - CTA to install prebuilt rules - at this moment, it's a link button with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. + - CTA to upgrade prebuilt rules - at this moment, it's a tab with a counter (implemented) and a callout with a link button (not yet implemented) on the Rule Management page. + ### Assumptions - Below scenarios only apply to prebuilt detection rules. @@ -44,16 +48,18 @@ Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule ## Scenarios -### Prebuilt rules package installation +### Package installation #### **Scenario: Package is installed via Fleet** **Coverage**: 0 tests - covered by other scenarios. This scenario will be covered by the **Users install the latest prebuilt rules** scenario, which will include 1 e2e test which installs the rules from the real package. +TODO: Where are scenarios for the real package? + ```Gherkin -Given user doesn't have the package installed -When they navigate to the Rule Management page -Then the package is installed in the background from EPR +Given the package is not installed +When user opens the Rule Management page +Then the package gets installed in the background from EPR ``` #### **Scenario: Package is installed via bundled Fleet package in Kibana** @@ -61,10 +67,10 @@ Then the package is installed in the background from EPR **Coverage**: 1 integration test. ```Gherkin -Given user doesn't have the package installed -And the user is in an air-gapped environment -When they navigate to the Rule Management page -Then the package is installed in the background from packages bundled into Kibana +Given the package is not installed +And user is in an air-gapped environment +When user opens the Rule Management page +Then the package gets installed in the background from packages bundled into Kibana ``` #### **Scenario: Large package can be installed on a small Kibana instance** @@ -72,86 +78,252 @@ Then the package is installed in the background from packages bundled into Kiban **Coverage**: 1 integration test. ```Gherkin -Given user doesn't have the package installed -And the package has the largest amount of historical rule versions installed (15000) +Given the package is not installed +And the package contains the largest amount of historical rule versions (15000) And the Kibana instance has a memory heap size of X Mb (see note below) -When they navigate to the Rule Management page +When user opens the Rule Management page Then the package is installed without Kibana crashing with an Out Of Memory error ``` +TODO: Update the `heap size of X Mb` to a concrete value and update the note below. + **Note**: The amount of memory is undefined as of now because, during implementation, we will try to find a memory heap threshold below which Kibana starts to crash constantly when you install the package with 15k rules. The plan is to then increase it to the point where it stops crashing, and use it as our value for this test. -### Notifications +### Rule installation and upgrade via the Prebuilt rules API -#### **Scenario: No callout messages are displayed when there are no pending installs/updates** +There's a legacy prebuilt rules API and a new one. Both should be tested against two types of the package: with and without historical rule versions. + +#### **Scenario: API can install all prebuilt rules** + +**Coverage**: 8 integration tests with mock rules: 4 examples below * 2 (we split checking API response and installed rules into two different tests). ```Gherkin -Given user has the latest version of prebuilt rules -When user navigates to the Rule Management page -Then no callout message is displayed for rules -And no badges with number of available rules to install/update are displayed +Given the package is installed +And the package contains N rules +When user installs all rules via install +Then the endpoint should return 200 with +And N rule objects should be created +And each rule object should have correct id and version Examples: - | prebuilt_rules_status | - | to install | - | to update | + | package_type | api | install_response | + | with historical versions | legacy | installed: N, updated: 0 | + | w/o historical versions | legacy | installed: N, updated: 0 | + | with historical versions | new | total: N, succeeded: N | + | w/o historical versions | new | total: N, succeeded: N | ``` -#### **Scenario: Callout message is displayed when there are new prebuilt rules available to install** +Notes: + +- Legacy API: + - install: `PUT /api/detection_engine/rules/prepackaged` +- New API: + - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` + +#### **Scenario: API can install prebuilt rules that are not yet installed** + +**Coverage**: 4 integration tests with mock rules. ```Gherkin -Given user already has 8.x prebuilt rules installed -And there are new prebuilt rules available to install -When user navigates to the Rules Management Page -Then user should see a callout message to install new prebuilt rules -And the number of new rules available to install should be displayed on the +Add Elastic Rules link +Given the package is installed +And the package contains N rules +When user installs all rules via install +And deletes one of the installed rules +And gets prebuilt rules status via status +Then the endpoint should return 200 with +When user installs all rules via install again +Then the endpoint should return 200 with + +Examples: + | package_type | api | status_response | install_response | + | with historical versions | legacy | not_installed: 1 | installed: 1, updated: 0 | + | w/o historical versions | legacy | not_installed: 1 | installed: 1, updated: 0 | + | with historical versions | new | to_install: 1 | total: 1, succeeded: 1 | + | w/o historical versions | new | to_install: 1 | total: 1, succeeded: 1 | ``` -#### **Scenario: Callout message is displayed when there are new updates on already installed prebuilt rules** +Notes: + +- Legacy API: + - install: `PUT /api/detection_engine/rules/prepackaged` + - status: `GET /api/detection_engine/rules/prepackaged/_status` +- New API: + - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` + - status: `GET /internal/detection_engine/prebuilt_rules/status` + +#### **Scenario: API can upgrade prebuilt rules that are outdated** + +**Coverage**: 4 integration tests with mock rules. ```Gherkin -Given user already has 8.x prebuilt rules installed -And there are new updates available for those prebuilt rules -When users navigate to the Rules Management Page -Then users should see an update callout message -And the number of outdated rules should be displayed on the Rules Updates tab +Given the package is installed +And the package contains N rules +When user installs all rules via install +And new X+1 version of a rule asset +And user gets prebuilt rules status via status +Then the endpoint should return 200 with +When user upgrades all rules via upgrade +Then the endpoint should return 200 with + +Examples: + | package_type | api | assets_update | status_response | upgrade_response | + | with historical versions | legacy | gets added | not_updated: 1 | installed: 0, updated: 1 | + | w/o historical versions | legacy | replaces X | not_updated: 1 | installed: 0, updated: 1 | + | with historical versions | new | gets added | to_upgrade: 1 | total: 1, succeeded: 1 | + | w/o historical versions | new | replaces X | to_upgrade: 1 | total: 1, succeeded: 1 | ``` -#### **Scenario: User is notified of available prebuilt rules to install when a rule is deleted** +TODO: Check why for the legacy API Dmitrii has added 2 integration tests for `rule package with historical versions` instead of 1: + +- `should update outdated prebuilt rules when previous historical versions available` +- `should update outdated prebuilt rules when previous historical versions unavailable` + +Notes: + +- Legacy API: + - install: `PUT /api/detection_engine/rules/prepackaged` + - upgrade: `PUT /api/detection_engine/rules/prepackaged` + - status: `GET /api/detection_engine/rules/prepackaged/_status` +- New API: + - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` + - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` + - status: `GET /internal/detection_engine/prebuilt_rules/status` + +#### **Scenario: API does not install or upgrade prebuilt rules if they are up to date** + +**Coverage**: 4 integration tests with mock rules. ```Gherkin -Given user has the latest version of prebuilt rules installed -When user navigates to Rules Management Page -And user deletes some prebuilt rules -Then user should see a callout message with the same amount of prebuilt rules ready to install -And the updated number of available rules to install should be displayed on the +Add Elastic Rules link +Given the package is installed +And the package contains N rules +When user installs all rules via install +And user gets prebuilt rules status via status +Then the endpoint should return 200 with +When user calls install +Then the endpoint should return 200 with +When user calls upgrade +Then the endpoint should return 200 with + +Examples: + | package_type | api | status_response | install_response | upgrade_response | + | with historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | + | w/o historical versions | legacy | not_installed: 0, not_updated: 0 | installed: 0, updated: 0 | installed: 0, updated: 0 | + | with historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | + | w/o historical versions | new | to_install: 0, to_upgrade: 0 | total: 0, succeeded: 0 | total: 0, succeeded: 0 | ``` -#### **Scenario: User is notified that all available rules have been installed in the Add Elastic Rules** +Notes: + +- Legacy API: + - install: `PUT /api/detection_engine/rules/prepackaged` + - upgrade: `PUT /api/detection_engine/rules/prepackaged` + - status: `GET /api/detection_engine/rules/prepackaged/_status` +- New API: + - install: `POST /internal/detection_engine/prebuilt_rules/installation/_perform` + - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` + - status: `GET /internal/detection_engine/prebuilt_rules/status` + +### Rule installation and upgrade notifications on the Rule Management page + +#### **Scenario: User is notified when no prebuilt rules are installed** + +**Coverage**: 1 e2e test with mock rules + 1 integration test. ```Gherkin -Given user has all available rules installed -When user navigates to the Add Elastic Rules Page -Then user should see a message indicating that all available rules have been installed -And user should see a CTA that leads to the Rules Management Page -And user should navigate back to Rules Management Page when clicking on the CTA +Given no prebuilt rules are installed in Kibana +And there are X prebuilt rules available to install +When user opens the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see a number of rules available to install (X) +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +And user should NOT see the Rule Updates table ``` -#### **Scenario: User is notified that all installed rules are up to date in the Rule Updates tab** +#### **Scenario: User is NOT notified when all prebuilt rules are installed and up to date** + +**Coverage**: 1 e2e test with mock rules + 1 integration test. ```Gherkin -Given user has all available rules installed -When user navigates to the Rule Update -Then user should see a message indicating that all installed rules are up to date +Given all the latest prebuilt rules are installed in Kibana +When user opens the Rule Management page +Then user should NOT see a CTA to install prebuilt rules +And user should NOT see a number of rules available to install +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +And user should NOT see the Rule Updates table ``` -### Prebuilt Rules Installation +#### **Scenario: User is notified when some prebuilt rules can be installed** + +**Coverage**: 1 e2e test with mock rules + 1 integration test. + +```Gherkin +Given X prebuilt rules are installed in Kibana +And there are Y more prebuilt rules available to install +And for all X installed rules there are no new versions available +When user opens the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see the number of rules available to install (Y) +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +And user should NOT see the Rule Updates table +``` + +#### **Scenario: User is notified when some prebuilt rules can be upgraded** + +**Coverage**: 1 e2e test with mock rules + 1 integration test. + +```Gherkin +Given X prebuilt rules are installed in Kibana +And there are no more prebuilt rules available to install +And for Z of the installed rules there are new versions available +When user opens the Rule Management page +Then user should NOT see a CTA to install prebuilt rules +And user should NOT see a number of rules available to install +And user should see a CTA to upgrade prebuilt rules +And user should see the number of rules available to upgrade (Z) +And user should see the Rule Updates table +``` + +#### **Scenario: User is notified when both rules to install and upgrade are available** + +**Coverage**: 1 e2e test with mock rules + 1 integration test. + +```Gherkin +Given X prebuilt rules are installed in Kibana +And there are Y more prebuilt rules available to install +And for Z of the installed rules there are new versions available +When user opens the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see the number of rules available to install (Y) +And user should see a CTA to upgrade prebuilt rules +And user should see the number of rules available to upgrade (Z) +And user should see the Rule Updates table +``` + +#### **Scenario: User is notified after a prebuilt rule gets deleted** + +**Coverage**: 1 e2e test with mock rules + 1 integration test. + +```Gherkin +Given X prebuilt rules are installed in Kibana +And there are no more prebuilt rules available to install +When user opens the Rule Management page +And user deletes Y prebuilt rules +Then user should see a CTA to install prebuilt rules +And user should see the number of rules available to install (Y) +``` + +### Rule installation workflow #### **Scenario: User without any installed prebuilt rules can install `` prebuilt rules** +**Coverage**: ?. + ```Gherkin Given a user that doesn’t have prebuilt rules installed -When user navigates to Add Elastic Rules Page +When user navigates to Add Rules page Then available prebuilt rules are displayed on Elastic Rules table And user can install prebuilt Rules And successfully installed message is displayed after installation @@ -165,22 +337,16 @@ Examples: | individual | ``` -#### **Scenario: User navigating to the Add Elastic Rules page sees a loading skeleton until the prebuilt rules package installation is completed** - -```Gherkin -Given a user that is on Rules Management Page -When user to the Add Elastic Rules page before rules package is installed -Then a loading skeleton is displayed until the installation is completed -``` - -### Prebuilt Rules Update +### Rule upgrade workflow #### **Scenario: Users can update prebuilt rules** +**Coverage**: ?. + ```Gherkin Given user already has 8.x prebuilt rules installed in Kibana And there are new updates available for those prebuilt rules -And user is on Rules Management Page +And user is on Rule Management page When user navigates to the Rules Update tab Then user should see all the prebuilt rules that have updates available And user can update outdated prebuilt rules @@ -188,16 +354,18 @@ And successfully updated message is displayed And Rules Upgrade tab counter is decreased according to the number of updated rules ``` -### Package Installation / Rule Installation / Rule Update Failure +### Package installation / rule installation / rule upgrade failure #### **Scenario: Error message is displayed when any prebuilt rules operation fails** +**Coverage**: ?. + ```Gherkin Given user is prebuilt rules When the installation or update process fails Then user should see an error message And prebuilt rules are not installed/updated -And the callout message for pending installs/updates is still displayed on Rules Management Page +And the callout message for pending installs/updates is still displayed on Rule Management page And the number of available rules to install and upgrade in the badges does not change Examples: @@ -212,21 +380,48 @@ Examples: #### **Scenario: No callout messages are displayed when rule package installation fails and no rules are avialble for install/update** +**Coverage**: ?. + ```Gherkin -Given user navigates to Rules Management Page +Given user navigates to Rule Management page And user is running a fresh instance And rule package installation fails Then no callouts message should be displayed And the number of available rules to install and upgrade in the badges does not change ``` -### Add Elastic Rules Page +### Add Rules page + +#### **Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed** + +**Coverage**: ?. + +```Gherkin +Given prebuilt rules package is not installed +When user opens the Add Rules page +Then user should see a loading skeleton until the package installation is completed +``` + +#### **Scenario: Empty screen is shown when all prebuilt rules are installed** -#### **Scenario: New workflow elements are displayed on Rules Management Page** +**Coverage**: 1 e2e test with mock rules. + +```Gherkin +Given user has all the available prebuilt rules installed in Kibana +When user opens the Add Rules page +Then user should see a message indicating that all available rules have been installed +And user should see a CTA that leads to the Rule Management page +When user clicks on the CTA +Then user should be navigated back to Rule Management page +``` + +#### **Scenario: New workflow elements are displayed on Rule Management page** + +**Coverage**: ?. ```Gherkin Given a user that doesn’t have `security_detection_engine` package installed -When user is on Rules Management Page +When user is on Rule Management page Then "+Add Elastic rules" menu with available Rules counter is displayed And Rule Updates tab is displayed And "+Add Elastic rules" button is displayed on empty Rules Table @@ -234,6 +429,8 @@ And "+Add Elastic rules" button is displayed on empty Rules Table #### **Scenario: Rules settings persist on Add Elastic Rules table** +**Coverage**: ?. + ```Gherkin Given a user has Rules listed on Add Elastic Rules page When @@ -248,14 +445,18 @@ Examples: #### **Scenario: User can navigate back to Rules Management page** +**Coverage**: ?. + ```Gherkin Given a user is on Add Rules Page When user navigates back to Rules Management page -Then Rules Management Page is properly displayed +Then Rule Management page is properly displayed ``` #### **Scenario: User can filter prebuilt rules by rule name or by tag** +**Coverage**: ?. + ```Gherkin Given a user is on Add Rules Page When user filters by @@ -267,10 +468,24 @@ Examples: | Tag filter | ``` -### Rule Updates tab +### Rule Updates table + +#### **Scenario: Empty screen is shown when all installed prebuilt rules are up to date** + +**Coverage**: ?. + +```Gherkin +Given user has some prebuilt rules installed in Kibana +And all of them are up to date (have the latest versions) +When user opens the Rule Management page +And selects the Rule Updates tab +Then user should see a message indicating that all installed rules are up to date +``` #### **Scenario: Rules settings persist on Rule Updates table** +**Coverage**: ?. + ```Gherkin Given a user has Rules listed on Rule Updates table When @@ -285,14 +500,18 @@ Examples: #### **Scenario: User can navigate back to Rules Management tab** +**Coverage**: ?. + ```Gherkin Given a user is on Rule Updates tab When user navigates back to Rules Management page -Then Rules Management Page is properly displayed +Then Rule Management page is properly displayed ``` #### **Scenario: User can filter prebuilt rules by rule name or by tag** +**Coverage**: ?. + ```Gherkin Given a user is on Rule Updates tab When user filters by @@ -308,18 +527,22 @@ Examples: #### **Scenario: User with read privileges on security solution cannot install prebuilt rules** +**Coverage**: ?. + ```Gherkin Given a user with Security: read privileges on Security solution -When user navigates to Add Elastic Rules Page +When user navigates to Add Rules page Then user can see available prebuilt rules to install And user cannot Install those prebuilt rules ``` #### **Scenario: User with read privileges on security solution cannot update prebuilt rules** +**Coverage**: ?. + ```Gherkin Given a user with Security: read privileges on Security solution -When user navigates to Rule Updates Tab on Rules Management Page +When user navigates to Rule Updates Tab on Rule Management page Then user can see new updates for installed prebuilt rules And user cannot Update those prebuilt rules ``` @@ -328,6 +551,8 @@ And user cannot Update those prebuilt rules #### **Scenario: User can operate with prebuilt rules when user upgrades from version `` to 8.9 version** +**Coverage**: ?. + ```Gherkin Given a user that is upgrading from version to version 8.9 And the instance contains already installed prebuilt rules diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/original_scenarios.md b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/original_scenarios.md new file mode 100644 index 0000000000000..7b3423f28fc2a --- /dev/null +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/original_scenarios.md @@ -0,0 +1,264 @@ +# Installation and Upgrade of Prebuilt Rules + +## Scenarios + +### Notifications + +#### **Scenario: No callout messages are displayed when there are no pending installs/updates** + +```Gherkin +Given user has the latest version of prebuilt rules +When user navigates to the Rule Management page +Then no callout message is displayed for rules +And no badges with number of available rules to install/update are displayed + +Examples: + | prebuilt_rules_status | + | to install | + | to update | +``` + +#### **Scenario: Callout message is displayed when there are new prebuilt rules available to install** + +```Gherkin +Given user already has 8.x prebuilt rules installed +And there are new prebuilt rules available to install +When user navigates to the Rules Management Page +Then user should see a callout message to install new prebuilt rules +And the number of new rules available to install should be displayed on the +Add Elastic Rules link +``` + +#### **Scenario: Callout message is displayed when there are new updates on already installed prebuilt rules** + +```Gherkin +Given user already has 8.x prebuilt rules installed +And there are new updates available for those prebuilt rules +When users navigate to the Rules Management Page +Then users should see an update callout message +And the number of outdated rules should be displayed on the Rules Updates tab +``` + +#### **Scenario: User is notified of available prebuilt rules to install when a rule is deleted** + +```Gherkin +Given user has the latest version of prebuilt rules installed +When user navigates to Rules Management Page +And user deletes some prebuilt rules +Then user should see a callout message with the same amount of prebuilt rules ready to install +And the updated number of available rules to install should be displayed on the +Add Elastic Rules link +``` + +#### **Scenario: User is notified that all available rules have been installed in the Add Elastic Rules** + +```Gherkin +Given user has all available rules installed +When user navigates to the Add Elastic Rules Page +Then user should see a message indicating that all available rules have been installed +And user should see a CTA that leads to the Rules Management Page +And user should navigate back to Rules Management Page when clicking on the CTA +``` + +#### **Scenario: User is notified that all installed rules are up to date in the Rule Updates tab** + +```Gherkin +Given user has all available rules installed +When user navigates to the Rule Update +Then user should see a message indicating that all installed rules are up to date +``` + +### Prebuilt Rules Installation + +#### **Scenario: User without any installed prebuilt rules can install `` prebuilt rules** + +```Gherkin +Given a user that doesn’t have prebuilt rules installed +When user navigates to Add Elastic Rules Page +Then available prebuilt rules are displayed on Elastic Rules table +And user can install prebuilt Rules +And successfully installed message is displayed after installation +And installed rules are removed from Elastic Rules table +And rules to install counter is decreased accordingly + +Examples: + | amount | + | all | + | selected | + | individual | +``` + +#### **Scenario: User navigating to the Add Elastic Rules page sees a loading skeleton until the prebuilt rules package installation is completed** + +```Gherkin +Given a user that is on Rules Management Page +When user to the Add Elastic Rules page before rules package is installed +Then a loading skeleton is displayed until the installation is completed +``` + +### Prebuilt Rules Update + +#### **Scenario: Users can update prebuilt rules** + +```Gherkin +Given user already has 8.x prebuilt rules installed in Kibana +And there are new updates available for those prebuilt rules +And user is on Rules Management Page +When user navigates to the Rules Update tab +Then user should see all the prebuilt rules that have updates available +And user can update outdated prebuilt rules +And successfully updated message is displayed +And Rules Upgrade tab counter is decreased according to the number of updated rules +``` + +### Package Installation / Rule Installation / Rule Update Failure + +#### **Scenario: Error message is displayed when any prebuilt rules operation fails** + +```Gherkin +Given user is prebuilt rules +When the installation or update process fails +Then user should see an error message +And prebuilt rules are not installed/updated +And the callout message for pending installs/updates is still displayed on Rules Management Page +And the number of available rules to install and upgrade in the badges does not change + +Examples: + | action | + | installing all | + | installing selected | + | installing individual | + | updating all | + | updating selected | + | updating individual | +``` + +#### **Scenario: No callout messages are displayed when rule package installation fails and no rules are avialble for install/update** + +```Gherkin +Given user navigates to Rules Management Page +And user is running a fresh instance +And rule package installation fails +Then no callouts message should be displayed +And the number of available rules to install and upgrade in the badges does not change +``` + +### Add Elastic Rules Page + +#### **Scenario: New workflow elements are displayed on Rules Management Page** + +```Gherkin +Given a user that doesn’t have `security_detection_engine` package installed +When user is on Rules Management Page +Then "+Add Elastic rules" menu with available Rules counter is displayed +And Rule Updates tab is displayed +And "+Add Elastic rules" button is displayed on empty Rules Table +``` + +#### **Scenario: Rules settings persist on Add Elastic Rules table** + +```Gherkin +Given a user has Rules listed on Add Elastic Rules page +When +Then the available rules state should persist for all the rules + +Examples: + | case | + | user reloads the page | + | after switching table pagination | + | after filtering and clear filters | +``` + +#### **Scenario: User can navigate back to Rules Management page** + +```Gherkin +Given a user is on Add Rules Page +When user navigates back to Rules Management page +Then Rules Management Page is properly displayed +``` + +#### **Scenario: User can filter prebuilt rules by rule name or by tag** + +```Gherkin +Given a user is on Add Rules Page +When user filters by +Then Add Rules Table is properly updated + +Examples: + | filter | + | rule name on search bar | + | Tag filter | +``` + +### Rule Updates tab + +#### **Scenario: Rules settings persist on Rule Updates table** + +```Gherkin +Given a user has Rules listed on Rule Updates table +When +Then the rules with available updates state should persist + +Examples: + | case | + | user reloads the page | + | after switching table pagination | + | after filtering and clear filters | +``` + +#### **Scenario: User can navigate back to Rules Management tab** + +```Gherkin +Given a user is on Rule Updates tab +When user navigates back to Rules Management page +Then Rules Management Page is properly displayed +``` + +#### **Scenario: User can filter prebuilt rules by rule name or by tag** + +```Gherkin +Given a user is on Rule Updates tab +When user filters by +Then Rule Updates tab is properly updated + +Examples: + | filter | + | rule name on search bar | + | Tag filter | +``` + +### Authorization / RBAC + +#### **Scenario: User with read privileges on security solution cannot install prebuilt rules** + +```Gherkin +Given a user with Security: read privileges on Security solution +When user navigates to Add Elastic Rules Page +Then user can see available prebuilt rules to install +And user cannot Install those prebuilt rules +``` + +#### **Scenario: User with read privileges on security solution cannot update prebuilt rules** + +```Gherkin +Given a user with Security: read privileges on Security solution +When user navigates to Rule Updates Tab on Rules Management Page +Then user can see new updates for installed prebuilt rules +And user cannot Update those prebuilt rules +``` + +### Kibana upgrade + +#### **Scenario: User can operate with prebuilt rules when user upgrades from version `` to 8.9 version** + +```Gherkin +Given a user that is upgrading from version to version 8.9 +And the instance contains already installed prebuilt rules +When the upgrade is complete +Then user can install new prebuilt rules +And remove installed prebuilt rules +And update prebuilt rules from to 8.9 + +Examples: + | version | + | 8.7 | + | 7.17.x | +``` From 8bf3404902971aeb4b2241810a1b600128264bc8 Mon Sep 17 00:00:00 2001 From: Georgii Gorbachev Date: Sun, 2 Jul 2023 01:21:36 +0200 Subject: [PATCH 09/13] More changes --- .../installation_and_upgrade.md | 314 +++++++++--------- 1 file changed, 163 insertions(+), 151 deletions(-) diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md index 1d34fa29c83a9..6b80256e88c62 100644 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md @@ -315,82 +315,85 @@ Then user should see a CTA to install prebuilt rules And user should see the number of rules available to install (Y) ``` -### Rule installation workflow +### Rule installation workflow: base cases -#### **Scenario: User without any installed prebuilt rules can install `` prebuilt rules** +#### **Scenario: User can install prebuilt rules one by one** -**Coverage**: ?. +**Coverage**: 1 e2e test with mock rules. ```Gherkin -Given a user that doesn’t have prebuilt rules installed -When user navigates to Add Rules page -Then available prebuilt rules are displayed on Elastic Rules table -And user can install prebuilt Rules -And successfully installed message is displayed after installation -And installed rules are removed from Elastic Rules table -And rules to install counter is decreased accordingly - -Examples: - | amount | - | all | - | selected | - | individual | +Given no prebuilt rules are installed in Kibana +And there are X prebuilt rules available to install +When user opens the Add Rules page +Then prebuilt rules available for installation should be displayed in the table +When user installs one individual rule +Then success message should be displayed after installation +And the installed rule should be removed from the table +When user navigates back to the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see the number of rules available to install decreased by 1 ``` -### Rule upgrade workflow - -#### **Scenario: Users can update prebuilt rules** +#### **Scenario: User can install multiple prebuilt rules selected on the page** -**Coverage**: ?. +**Coverage**: 1 e2e test with mock rules. ```Gherkin -Given user already has 8.x prebuilt rules installed in Kibana -And there are new updates available for those prebuilt rules -And user is on Rule Management page -When user navigates to the Rules Update tab -Then user should see all the prebuilt rules that have updates available -And user can update outdated prebuilt rules -And successfully updated message is displayed -And Rules Upgrade tab counter is decreased according to the number of updated rules -``` +Given no prebuilt rules are installed in Kibana +And there are X prebuilt rules available to install +When user opens the Add Rules page +Then prebuilt rules available for installation should be displayed in the table +When user selects rules +Then user should see a CTA to install number of rules +When user clicks the CTA +Then success message should be displayed after installation +And all the installed rules should be removed from the table +When user navigates back to the Rule Management page +Then user should see a CTA to install prebuilt rules +And user should see the number of rules available to install decreased by number of installed rules -### Package installation / rule installation / rule upgrade failure +Examples: + | Y | + | a few rules on the page, e.g. 2 | + | all rules on the page, e.g. 12 | +``` -#### **Scenario: Error message is displayed when any prebuilt rules operation fails** +#### **Scenario: User can install all available prebuilt rules at once** -**Coverage**: ?. +**Coverage**: 1 e2e test with mock rules. ```Gherkin -Given user is prebuilt rules -When the installation or update process fails -Then user should see an error message -And prebuilt rules are not installed/updated -And the callout message for pending installs/updates is still displayed on Rule Management page -And the number of available rules to install and upgrade in the badges does not change - -Examples: - | action | - | installing all | - | installing selected | - | installing individual | - | updating all | - | updating selected | - | updating individual | +Given no prebuilt rules are installed in Kibana +And there are X prebuilt rules available to install +When user opens the Add Rules page +Then prebuilt rules available for installation should be displayed in the table +When user installs all rules +Then success message should be displayed after installation +And all the rules should be removed from the table +And user should see a message indicating that all available rules have been installed +And user should see a CTA that leads to the Rule Management page +When user clicks on the CTA +Then user should be navigated back to Rule Management page +And user should NOT see a CTA to install prebuilt rules +And user should NOT see a number of rules available to install ``` -#### **Scenario: No callout messages are displayed when rule package installation fails and no rules are avialble for install/update** +#### **Scenario: Empty screen is shown when all prebuilt rules are installed** -**Coverage**: ?. +**Coverage**: 1 e2e test with mock rules. ```Gherkin -Given user navigates to Rule Management page -And user is running a fresh instance -And rule package installation fails -Then no callouts message should be displayed -And the number of available rules to install and upgrade in the badges does not change +Given all the available prebuilt rules are installed in Kibana +When user opens the Add Rules page +Then user should see a message indicating that all available rules have been installed +And user should see a CTA that leads to the Rule Management page ``` -### Add Rules page +### Rule installation workflow: filtering, sorting, pagination + +TODO: add scenarios + +### Rule installation workflow: misc cases #### **Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed** @@ -402,167 +405,176 @@ When user opens the Add Rules page Then user should see a loading skeleton until the package installation is completed ``` -#### **Scenario: Empty screen is shown when all prebuilt rules are installed** +#### **Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs** -**Coverage**: 1 e2e test with mock rules. +**Coverage**: ?. ```Gherkin -Given user has all the available prebuilt rules installed in Kibana -When user opens the Add Rules page -Then user should see a message indicating that all available rules have been installed -And user should see a CTA that leads to the Rule Management page -When user clicks on the CTA -Then user should be navigated back to Rule Management page +Given user is on the Add Rules page +When user navigates to the Rule Management page via breadcrumbs +Then the Rule Management page should be displayed ``` -#### **Scenario: New workflow elements are displayed on Rule Management page** +### Rule upgrade workflow: base cases -**Coverage**: ?. +#### **Scenario: User can upgrade prebuilt rules one by one** + +**Coverage**: 1 e2e test with mock rules. ```Gherkin -Given a user that doesn’t have `security_detection_engine` package installed -When user is on Rule Management page -Then "+Add Elastic rules" menu with available Rules counter is displayed -And Rule Updates tab is displayed -And "+Add Elastic rules" button is displayed on empty Rules Table +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user upgrades one individual rule +Then success message should be displayed after upgrade +And the upgraded rule should be removed from the table +And user should see the number of rules available to upgrade decreased by 1 ``` -#### **Scenario: Rules settings persist on Add Elastic Rules table** +#### **Scenario: User can upgrade multiple prebuilt rules selected on the page** -**Coverage**: ?. +**Coverage**: 1 e2e test with mock rules. ```Gherkin -Given a user has Rules listed on Add Elastic Rules page -When -Then the available rules state should persist for all the rules +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user selects rules +Then user should see a CTA to upgrade number of rules +When user clicks the CTA +Then success message should be displayed after upgrade +And all the upgraded rules should be removed from the table +And user should see the number of rules available to upgrade decreased by number of upgraded rules Examples: - | case | - | user reloads the page | - | after switching table pagination | - | after filtering and clear filters | + | Z | + | a few rules on the page, e.g. 2 | + | all rules on the page, e.g. 12 | ``` -#### **Scenario: User can navigate back to Rules Management page** +#### **Scenario: User can upgrade all available prebuilt rules at once** -**Coverage**: ?. +**Coverage**: 1 e2e test with mock rules. ```Gherkin -Given a user is on Add Rules Page -When user navigates back to Rules Management page -Then Rule Management page is properly displayed +Given X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +And user is on the Rule Management page +When user opens the Rule Updates table +Then Y rules available for upgrade should be displayed in the table +When user upgrades all rules +Then success message should be displayed after upgrade +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +And user should NOT see the Rule Updates table ``` -#### **Scenario: User can filter prebuilt rules by rule name or by tag** +### Rule upgrade workflow: filtering, sorting, pagination -**Coverage**: ?. +TODO: add scenarios -```Gherkin -Given a user is on Add Rules Page -When user filters by -Then Add Rules Table is properly updated +### Rule upgrade workflow: misc cases -Examples: - | filter | - | rule name on search bar | - | Tag filter | -``` - -### Rule Updates table - -#### **Scenario: Empty screen is shown when all installed prebuilt rules are up to date** +#### **Scenario: User opening the Rule Updates table sees a loading skeleton until the package installation is completed** **Coverage**: ?. ```Gherkin -Given user has some prebuilt rules installed in Kibana -And all of them are up to date (have the latest versions) +Given prebuilt rules package is not installed When user opens the Rule Management page -And selects the Rule Updates tab -Then user should see a message indicating that all installed rules are up to date +And user opens the Rule Updates table +Then user should see a loading skeleton until the package installation is completed ``` -#### **Scenario: Rules settings persist on Rule Updates table** +### Error handling -**Coverage**: ?. - -```Gherkin -Given a user has Rules listed on Rule Updates table -When -Then the rules with available updates state should persist - -Examples: - | case | - | user reloads the page | - | after switching table pagination | - | after filtering and clear filters | -``` - -#### **Scenario: User can navigate back to Rules Management tab** +#### **Scenario: Error is handled when the package installation fails** **Coverage**: ?. ```Gherkin -Given a user is on Rule Updates tab -When user navigates back to Rules Management page -Then Rule Management page is properly displayed +Given the package is not installed +And no prebuilt rules are installed in Kibana +When user opens the Rule Management page +And the package installation fails +Then user should NOT see a CTA to install prebuilt rules +And user should NOT see a number of rules available to install +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +And user should NOT see the Rule Updates table ``` -#### **Scenario: User can filter prebuilt rules by rule name or by tag** +#### **Scenario: Error is handled when any operation on prebuilt rules fails** **Coverage**: ?. ```Gherkin -Given a user is on Rule Updates tab -When user filters by -Then Rule Updates tab is properly updated +Given user is prebuilt rules +When the operation fails +Then user should see an error message +And prebuilt rules should not be installed/upgraded +And the CTAs to install/upgrade prebuilt rules should remain on the Rule Management page +And the numbers of available rules to install/upgrade should not change Examples: - | filter | - | rule name on search bar | - | Tag filter | + | operation | + | installing all | + | installing selected | + | installing individual | + | upgrading all | + | upgrading selected | + | upgrading individual | ``` ### Authorization / RBAC -#### **Scenario: User with read privileges on security solution cannot install prebuilt rules** +#### **Scenario: User with read privileges on Security Solution cannot install prebuilt rules** -**Coverage**: ?. +**Coverage**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and installation endpoints. ```Gherkin -Given a user with Security: read privileges on Security solution -When user navigates to Add Rules page -Then user can see available prebuilt rules to install -And user cannot Install those prebuilt rules +Given user with "Security: read" privileges on Security Solution +And no prebuilt rules are installed in Kibana +And there are prebuilt rules available to install +When user opens the Add Rules page +Then user should see prebuilt rules available to install +But user should not be able to install them ``` -#### **Scenario: User with read privileges on security solution cannot update prebuilt rules** +#### **Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules** -**Coverage**: ?. +**Coverage**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and upgrade endpoints. ```Gherkin -Given a user with Security: read privileges on Security solution -When user navigates to Rule Updates Tab on Rule Management page -Then user can see new updates for installed prebuilt rules -And user cannot Update those prebuilt rules +Given user with "Security: read" privileges on Security Solution +And X prebuilt rules are installed in Kibana +And for Y of the installed rules there are new versions available +When user opens the Rule Management page +And user opens the Rule Updates table +Then user should see prebuilt rules available to upgrade +But user should not be able to upgrade them ``` ### Kibana upgrade -#### **Scenario: User can operate with prebuilt rules when user upgrades from version `` to 8.9 version** +#### **Scenario: User can use prebuilt rules after upgrading Kibana from version A to B** -**Coverage**: ?. +**Coverage**: not covered, manual testing required. ```Gherkin -Given a user that is upgrading from version to version 8.9 -And the instance contains already installed prebuilt rules +Given user is upgrading Kibana from version to version +And the instance contains already installed prebuilt rules When the upgrade is complete -Then user can install new prebuilt rules -And remove installed prebuilt rules -And update prebuilt rules from to 8.9 +Then user should be able to install new prebuilt rules +And delete installed prebuilt rules +And upgrade installed prebuilt rules that have newer versions in Examples: - | version | - | 8.7 | - | 7.17.x | + | A | B | + | 8.7 | 8.9.0 | + | 7.17.x | 8.9.0 | ``` From 318168012130cfbd511aecf72ad8bfa42088a2f4 Mon Sep 17 00:00:00 2001 From: Georgii Gorbachev Date: Sun, 2 Jul 2023 02:45:54 +0200 Subject: [PATCH 10/13] Rename coverage to automation --- .../installation_and_upgrade.md | 56 +++++++++---------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md index 6b80256e88c62..5ce2f524b3210 100644 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md @@ -52,7 +52,7 @@ Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule #### **Scenario: Package is installed via Fleet** -**Coverage**: 0 tests - covered by other scenarios. This scenario will be covered by the **Users install the latest prebuilt rules** scenario, which will include 1 e2e test which installs the rules from the real package. +**Automation**: 0 tests - covered by other scenarios. This scenario will be covered by the **Users install the latest prebuilt rules** scenario, which will include 1 e2e test which installs the rules from the real package. TODO: Where are scenarios for the real package? @@ -64,7 +64,7 @@ Then the package gets installed in the background from EPR #### **Scenario: Package is installed via bundled Fleet package in Kibana** -**Coverage**: 1 integration test. +**Automation**: 1 integration test. ```Gherkin Given the package is not installed @@ -75,7 +75,7 @@ Then the package gets installed in the background from packages bundled into Kib #### **Scenario: Large package can be installed on a small Kibana instance** -**Coverage**: 1 integration test. +**Automation**: 1 integration test. ```Gherkin Given the package is not installed @@ -95,7 +95,7 @@ There's a legacy prebuilt rules API and a new one. Both should be tested against #### **Scenario: API can install all prebuilt rules** -**Coverage**: 8 integration tests with mock rules: 4 examples below * 2 (we split checking API response and installed rules into two different tests). +**Automation**: 8 integration tests with mock rules: 4 examples below * 2 (we split checking API response and installed rules into two different tests). ```Gherkin Given the package is installed @@ -122,7 +122,7 @@ Notes: #### **Scenario: API can install prebuilt rules that are not yet installed** -**Coverage**: 4 integration tests with mock rules. +**Automation**: 4 integration tests with mock rules. ```Gherkin Given the package is installed @@ -153,7 +153,7 @@ Notes: #### **Scenario: API can upgrade prebuilt rules that are outdated** -**Coverage**: 4 integration tests with mock rules. +**Automation**: 4 integration tests with mock rules. ```Gherkin Given the package is installed @@ -191,7 +191,7 @@ Notes: #### **Scenario: API does not install or upgrade prebuilt rules if they are up to date** -**Coverage**: 4 integration tests with mock rules. +**Automation**: 4 integration tests with mock rules. ```Gherkin Given the package is installed @@ -227,7 +227,7 @@ Notes: #### **Scenario: User is notified when no prebuilt rules are installed** -**Coverage**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test. ```Gherkin Given no prebuilt rules are installed in Kibana @@ -242,7 +242,7 @@ And user should NOT see the Rule Updates table #### **Scenario: User is NOT notified when all prebuilt rules are installed and up to date** -**Coverage**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test. ```Gherkin Given all the latest prebuilt rules are installed in Kibana @@ -256,7 +256,7 @@ And user should NOT see the Rule Updates table #### **Scenario: User is notified when some prebuilt rules can be installed** -**Coverage**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -272,7 +272,7 @@ And user should NOT see the Rule Updates table #### **Scenario: User is notified when some prebuilt rules can be upgraded** -**Coverage**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -288,7 +288,7 @@ And user should see the Rule Updates table #### **Scenario: User is notified when both rules to install and upgrade are available** -**Coverage**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -304,7 +304,7 @@ And user should see the Rule Updates table #### **Scenario: User is notified after a prebuilt rule gets deleted** -**Coverage**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -319,7 +319,7 @@ And user should see the number of rules available to install (Y) #### **Scenario: User can install prebuilt rules one by one** -**Coverage**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules. ```Gherkin Given no prebuilt rules are installed in Kibana @@ -336,7 +336,7 @@ And user should see the number of rules available to install decreased by 1 #### **Scenario: User can install multiple prebuilt rules selected on the page** -**Coverage**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules. ```Gherkin Given no prebuilt rules are installed in Kibana @@ -360,7 +360,7 @@ Examples: #### **Scenario: User can install all available prebuilt rules at once** -**Coverage**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules. ```Gherkin Given no prebuilt rules are installed in Kibana @@ -380,7 +380,7 @@ And user should NOT see a number of rules available to install #### **Scenario: Empty screen is shown when all prebuilt rules are installed** -**Coverage**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules. ```Gherkin Given all the available prebuilt rules are installed in Kibana @@ -397,7 +397,7 @@ TODO: add scenarios #### **Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed** -**Coverage**: ?. +**Automation**: ?. ```Gherkin Given prebuilt rules package is not installed @@ -407,7 +407,7 @@ Then user should see a loading skeleton until the package installation is comple #### **Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs** -**Coverage**: ?. +**Automation**: ?. ```Gherkin Given user is on the Add Rules page @@ -419,7 +419,7 @@ Then the Rule Management page should be displayed #### **Scenario: User can upgrade prebuilt rules one by one** -**Coverage**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -435,7 +435,7 @@ And user should see the number of rules available to upgrade decreased by 1 #### **Scenario: User can upgrade multiple prebuilt rules selected on the page** -**Coverage**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -458,7 +458,7 @@ Examples: #### **Scenario: User can upgrade all available prebuilt rules at once** -**Coverage**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -481,7 +481,7 @@ TODO: add scenarios #### **Scenario: User opening the Rule Updates table sees a loading skeleton until the package installation is completed** -**Coverage**: ?. +**Automation**: ?. ```Gherkin Given prebuilt rules package is not installed @@ -494,7 +494,7 @@ Then user should see a loading skeleton until the package installation is comple #### **Scenario: Error is handled when the package installation fails** -**Coverage**: ?. +**Automation**: ?. ```Gherkin Given the package is not installed @@ -510,7 +510,7 @@ And user should NOT see the Rule Updates table #### **Scenario: Error is handled when any operation on prebuilt rules fails** -**Coverage**: ?. +**Automation**: ?. ```Gherkin Given user is prebuilt rules @@ -534,7 +534,7 @@ Examples: #### **Scenario: User with read privileges on Security Solution cannot install prebuilt rules** -**Coverage**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and installation endpoints. +**Automation**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and installation endpoints. ```Gherkin Given user with "Security: read" privileges on Security Solution @@ -547,7 +547,7 @@ But user should not be able to install them #### **Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules** -**Coverage**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and upgrade endpoints. +**Automation**: 1 e2e test with mock rules + 3 integration tests with mock rules for the status and upgrade endpoints. ```Gherkin Given user with "Security: read" privileges on Security Solution @@ -563,7 +563,7 @@ But user should not be able to upgrade them #### **Scenario: User can use prebuilt rules after upgrading Kibana from version A to B** -**Coverage**: not covered, manual testing required. +**Automation**: not automated, manual testing required. ```Gherkin Given user is upgrading Kibana from version to version From 939ea3f79ab4df9fb7f5e82edad62155742f86f6 Mon Sep 17 00:00:00 2001 From: Georgii Gorbachev Date: Mon, 3 Jul 2023 13:34:57 +0200 Subject: [PATCH 11/13] More changes after review --- .../installation_and_upgrade.md | 119 ++++++++++-------- 1 file changed, 70 insertions(+), 49 deletions(-) diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md index 5ce2f524b3210..86db1f2bb8ef5 100644 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md @@ -52,9 +52,7 @@ Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule #### **Scenario: Package is installed via Fleet** -**Automation**: 0 tests - covered by other scenarios. This scenario will be covered by the **Users install the latest prebuilt rules** scenario, which will include 1 e2e test which installs the rules from the real package. - -TODO: Where are scenarios for the real package? +**Automation**: 2 e2e tests that install the real package. ```Gherkin Given the package is not installed @@ -80,14 +78,12 @@ Then the package gets installed in the background from packages bundled into Kib ```Gherkin Given the package is not installed And the package contains the largest amount of historical rule versions (15000) -And the Kibana instance has a memory heap size of X Mb (see note below) +And the Kibana instance has a memory heap size of 700 Mb (see note below) When user opens the Rule Management page Then the package is installed without Kibana crashing with an Out Of Memory error ``` -TODO: Update the `heap size of X Mb` to a concrete value and update the note below. - -**Note**: The amount of memory is undefined as of now because, during implementation, we will try to find a memory heap threshold below which Kibana starts to crash constantly when you install the package with 15k rules. The plan is to then increase it to the point where it stops crashing, and use it as our value for this test. +**Note**: 600 Mb seems to always crash Kibana with an OOM error. 700 Mb runs with no issues in the Flaky test runner with 100 iterations: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/2215. ### Rule installation and upgrade via the Prebuilt rules API @@ -223,11 +219,41 @@ Notes: - upgrade: `POST /internal/detection_engine/prebuilt_rules/upgrade/_perform` - status: `GET /internal/detection_engine/prebuilt_rules/status` +### Scenarios for the real package + +#### **Scenario: User can install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new pckage** + +**Automation**: 1 integration test with real packages. + +```Gherkin +Given there are two package versions: N-1 and N +And the package of N-1 version is installed +When user calls the status endpoint +Then it should return a 200 response with some number of rules to install and 0 rules to upgrade +When user calls the installation/_review endpoint +Then it should return a 200 response matching the response of the status endpoint +When user calls the installation/_perform_ endpoint +Then it should return a 200 response matching the response of the status endpoint +And rules returned in this response should exist as alert saved objects +When user installs the package of N version +Then it should be installed successfully +When user calls the status endpoint +Then it should return a 200 response with some number of new rules to install and some number of rules to upgrade +When user calls the installation/_review endpoint +Then it should return a 200 response matching the response of the status endpoint +When user calls the installation/_perform_ endpoint +Then rules returned in this response should exist as alert saved objects +When user calls the upgrade/_review endpoint +Then it should return a 200 response matching the response of the status endpoint +When user calls the upgrade/_perform_ endpoint +Then rules returned in this response should exist as alert saved objects +``` + ### Rule installation and upgrade notifications on the Rule Management page -#### **Scenario: User is notified when no prebuilt rules are installed** +#### **Scenario: User is notified when no prebuilt rules are installed and there are rules available to install** -**Automation**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin Given no prebuilt rules are installed in Kibana @@ -240,9 +266,24 @@ And user should NOT see a number of rules available to upgrade And user should NOT see the Rule Updates table ``` +#### **Scenario: User is NOT notified when no prebuilt rules are installed and there are no prebuilt rules assets** + +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. + +```Gherkin +Given no prebuilt rules are installed in Kibana +And no prebuilt rule assets exist +When user opens the Rule Management page +Then user should NOT see a CTA to install prebuilt rules +And user should NOT see a number of rules available to install +And user should NOT see a CTA to upgrade prebuilt rules +And user should NOT see a number of rules available to upgrade +And user should NOT see the Rule Updates table +``` + #### **Scenario: User is NOT notified when all prebuilt rules are installed and up to date** -**Automation**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin Given all the latest prebuilt rules are installed in Kibana @@ -256,7 +297,7 @@ And user should NOT see the Rule Updates table #### **Scenario: User is notified when some prebuilt rules can be installed** -**Automation**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -272,7 +313,7 @@ And user should NOT see the Rule Updates table #### **Scenario: User is notified when some prebuilt rules can be upgraded** -**Automation**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -288,7 +329,7 @@ And user should see the Rule Updates table #### **Scenario: User is notified when both rules to install and upgrade are available** -**Automation**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -304,7 +345,7 @@ And user should see the Rule Updates table #### **Scenario: User is notified after a prebuilt rule gets deleted** -**Automation**: 1 e2e test with mock rules + 1 integration test. +**Automation**: 1 e2e test with mock rules + 1 integration test with mock rules for the /status endpoint. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -319,7 +360,7 @@ And user should see the number of rules available to install (Y) #### **Scenario: User can install prebuilt rules one by one** -**Automation**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/* endpoints in integration. ```Gherkin Given no prebuilt rules are installed in Kibana @@ -336,7 +377,7 @@ And user should see the number of rules available to install decreased by 1 #### **Scenario: User can install multiple prebuilt rules selected on the page** -**Automation**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/* endpoints in integration. ```Gherkin Given no prebuilt rules are installed in Kibana @@ -360,7 +401,7 @@ Examples: #### **Scenario: User can install all available prebuilt rules at once** -**Automation**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /installation/* endpoints in integration. ```Gherkin Given no prebuilt rules are installed in Kibana @@ -380,7 +421,7 @@ And user should NOT see a number of rules available to install #### **Scenario: Empty screen is shown when all prebuilt rules are installed** -**Automation**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules + 1 integration test. ```Gherkin Given all the available prebuilt rules are installed in Kibana @@ -397,7 +438,7 @@ TODO: add scenarios #### **Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed** -**Automation**: ?. +**Automation**: unit tests. ```Gherkin Given prebuilt rules package is not installed @@ -407,7 +448,7 @@ Then user should see a loading skeleton until the package installation is comple #### **Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs** -**Automation**: ?. +**Automation**: 1 e2e test. ```Gherkin Given user is on the Add Rules page @@ -419,7 +460,7 @@ Then the Rule Management page should be displayed #### **Scenario: User can upgrade prebuilt rules one by one** -**Automation**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/* endpoints in integration. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -435,7 +476,7 @@ And user should see the number of rules available to upgrade decreased by 1 #### **Scenario: User can upgrade multiple prebuilt rules selected on the page** -**Automation**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/* endpoints in integration. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -458,7 +499,7 @@ Examples: #### **Scenario: User can upgrade all available prebuilt rules at once** -**Automation**: 1 e2e test with mock rules. +**Automation**: 1 e2e test with mock rules + integration tests with mock rules that would test /status and /upgrade/* endpoints in integration. ```Gherkin Given X prebuilt rules are installed in Kibana @@ -479,46 +520,26 @@ TODO: add scenarios ### Rule upgrade workflow: misc cases -#### **Scenario: User opening the Rule Updates table sees a loading skeleton until the package installation is completed** +#### **Scenario: User don't see the Rule Updates tab until the package installation is completed** -**Automation**: ?. +**Automation**: unit tests. ```Gherkin Given prebuilt rules package is not installed When user opens the Rule Management page -And user opens the Rule Updates table -Then user should see a loading skeleton until the package installation is completed +Then user should NOT see the Rule Updates tab until the package installation is completed and there are rules available for upgrade ``` ### Error handling -#### **Scenario: Error is handled when the package installation fails** - -**Automation**: ?. - -```Gherkin -Given the package is not installed -And no prebuilt rules are installed in Kibana -When user opens the Rule Management page -And the package installation fails -Then user should NOT see a CTA to install prebuilt rules -And user should NOT see a number of rules available to install -And user should NOT see a CTA to upgrade prebuilt rules -And user should NOT see a number of rules available to upgrade -And user should NOT see the Rule Updates table -``` - #### **Scenario: Error is handled when any operation on prebuilt rules fails** -**Automation**: ?. +**Automation**: unit tests. ```Gherkin -Given user is prebuilt rules -When the operation fails +When user is prebuilt rules +And this operation fails Then user should see an error message -And prebuilt rules should not be installed/upgraded -And the CTAs to install/upgrade prebuilt rules should remain on the Rule Management page -And the numbers of available rules to install/upgrade should not change Examples: | operation | From 52d45fbf320d7bac8b876db94d0ebf5f93cac53f Mon Sep 17 00:00:00 2001 From: Georgii Gorbachev Date: Tue, 4 Jul 2023 18:56:48 +0200 Subject: [PATCH 12/13] Final fixes and cleanup --- .../installation_and_upgrade.md | 3 +- .../prebuilt_rules/original_scenarios.md | 264 ------------------ 2 files changed, 1 insertion(+), 266 deletions(-) delete mode 100644 x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/original_scenarios.md diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md index 86db1f2bb8ef5..f07c134e37110 100644 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md +++ b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md @@ -32,7 +32,6 @@ Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule ### Assumptions - Below scenarios only apply to prebuilt detection rules. -- Most of our users are on the 7.17.x version, that’s why the 8.x version is specified on scenarios, because this test plan is considering a minimum version of 8.x. - Users should be able to install and upgrade prebuilt rules on the `Basic` license and higher. - EPR is available for fetching the package unless explicitly indicated otherwise. - Only the latest **stable** package is checked for installation/upgrade and pre-release packages are ignored. @@ -520,7 +519,7 @@ TODO: add scenarios ### Rule upgrade workflow: misc cases -#### **Scenario: User don't see the Rule Updates tab until the package installation is completed** +#### **Scenario: User doesn't see the Rule Updates tab until the package installation is completed** **Automation**: unit tests. diff --git a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/original_scenarios.md b/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/original_scenarios.md deleted file mode 100644 index 7b3423f28fc2a..0000000000000 --- a/x-pack/plugins/security_solution/cypress/test_plans/detection_response/prebuilt_rules/original_scenarios.md +++ /dev/null @@ -1,264 +0,0 @@ -# Installation and Upgrade of Prebuilt Rules - -## Scenarios - -### Notifications - -#### **Scenario: No callout messages are displayed when there are no pending installs/updates** - -```Gherkin -Given user has the latest version of prebuilt rules -When user navigates to the Rule Management page -Then no callout message is displayed for rules -And no badges with number of available rules to install/update are displayed - -Examples: - | prebuilt_rules_status | - | to install | - | to update | -``` - -#### **Scenario: Callout message is displayed when there are new prebuilt rules available to install** - -```Gherkin -Given user already has 8.x prebuilt rules installed -And there are new prebuilt rules available to install -When user navigates to the Rules Management Page -Then user should see a callout message to install new prebuilt rules -And the number of new rules available to install should be displayed on the +Add Elastic Rules link -``` - -#### **Scenario: Callout message is displayed when there are new updates on already installed prebuilt rules** - -```Gherkin -Given user already has 8.x prebuilt rules installed -And there are new updates available for those prebuilt rules -When users navigate to the Rules Management Page -Then users should see an update callout message -And the number of outdated rules should be displayed on the Rules Updates tab -``` - -#### **Scenario: User is notified of available prebuilt rules to install when a rule is deleted** - -```Gherkin -Given user has the latest version of prebuilt rules installed -When user navigates to Rules Management Page -And user deletes some prebuilt rules -Then user should see a callout message with the same amount of prebuilt rules ready to install -And the updated number of available rules to install should be displayed on the +Add Elastic Rules link -``` - -#### **Scenario: User is notified that all available rules have been installed in the Add Elastic Rules** - -```Gherkin -Given user has all available rules installed -When user navigates to the Add Elastic Rules Page -Then user should see a message indicating that all available rules have been installed -And user should see a CTA that leads to the Rules Management Page -And user should navigate back to Rules Management Page when clicking on the CTA -``` - -#### **Scenario: User is notified that all installed rules are up to date in the Rule Updates tab** - -```Gherkin -Given user has all available rules installed -When user navigates to the Rule Update -Then user should see a message indicating that all installed rules are up to date -``` - -### Prebuilt Rules Installation - -#### **Scenario: User without any installed prebuilt rules can install `` prebuilt rules** - -```Gherkin -Given a user that doesn’t have prebuilt rules installed -When user navigates to Add Elastic Rules Page -Then available prebuilt rules are displayed on Elastic Rules table -And user can install prebuilt Rules -And successfully installed message is displayed after installation -And installed rules are removed from Elastic Rules table -And rules to install counter is decreased accordingly - -Examples: - | amount | - | all | - | selected | - | individual | -``` - -#### **Scenario: User navigating to the Add Elastic Rules page sees a loading skeleton until the prebuilt rules package installation is completed** - -```Gherkin -Given a user that is on Rules Management Page -When user to the Add Elastic Rules page before rules package is installed -Then a loading skeleton is displayed until the installation is completed -``` - -### Prebuilt Rules Update - -#### **Scenario: Users can update prebuilt rules** - -```Gherkin -Given user already has 8.x prebuilt rules installed in Kibana -And there are new updates available for those prebuilt rules -And user is on Rules Management Page -When user navigates to the Rules Update tab -Then user should see all the prebuilt rules that have updates available -And user can update outdated prebuilt rules -And successfully updated message is displayed -And Rules Upgrade tab counter is decreased according to the number of updated rules -``` - -### Package Installation / Rule Installation / Rule Update Failure - -#### **Scenario: Error message is displayed when any prebuilt rules operation fails** - -```Gherkin -Given user is prebuilt rules -When the installation or update process fails -Then user should see an error message -And prebuilt rules are not installed/updated -And the callout message for pending installs/updates is still displayed on Rules Management Page -And the number of available rules to install and upgrade in the badges does not change - -Examples: - | action | - | installing all | - | installing selected | - | installing individual | - | updating all | - | updating selected | - | updating individual | -``` - -#### **Scenario: No callout messages are displayed when rule package installation fails and no rules are avialble for install/update** - -```Gherkin -Given user navigates to Rules Management Page -And user is running a fresh instance -And rule package installation fails -Then no callouts message should be displayed -And the number of available rules to install and upgrade in the badges does not change -``` - -### Add Elastic Rules Page - -#### **Scenario: New workflow elements are displayed on Rules Management Page** - -```Gherkin -Given a user that doesn’t have `security_detection_engine` package installed -When user is on Rules Management Page -Then "+Add Elastic rules" menu with available Rules counter is displayed -And Rule Updates tab is displayed -And "+Add Elastic rules" button is displayed on empty Rules Table -``` - -#### **Scenario: Rules settings persist on Add Elastic Rules table** - -```Gherkin -Given a user has Rules listed on Add Elastic Rules page -When -Then the available rules state should persist for all the rules - -Examples: - | case | - | user reloads the page | - | after switching table pagination | - | after filtering and clear filters | -``` - -#### **Scenario: User can navigate back to Rules Management page** - -```Gherkin -Given a user is on Add Rules Page -When user navigates back to Rules Management page -Then Rules Management Page is properly displayed -``` - -#### **Scenario: User can filter prebuilt rules by rule name or by tag** - -```Gherkin -Given a user is on Add Rules Page -When user filters by -Then Add Rules Table is properly updated - -Examples: - | filter | - | rule name on search bar | - | Tag filter | -``` - -### Rule Updates tab - -#### **Scenario: Rules settings persist on Rule Updates table** - -```Gherkin -Given a user has Rules listed on Rule Updates table -When -Then the rules with available updates state should persist - -Examples: - | case | - | user reloads the page | - | after switching table pagination | - | after filtering and clear filters | -``` - -#### **Scenario: User can navigate back to Rules Management tab** - -```Gherkin -Given a user is on Rule Updates tab -When user navigates back to Rules Management page -Then Rules Management Page is properly displayed -``` - -#### **Scenario: User can filter prebuilt rules by rule name or by tag** - -```Gherkin -Given a user is on Rule Updates tab -When user filters by -Then Rule Updates tab is properly updated - -Examples: - | filter | - | rule name on search bar | - | Tag filter | -``` - -### Authorization / RBAC - -#### **Scenario: User with read privileges on security solution cannot install prebuilt rules** - -```Gherkin -Given a user with Security: read privileges on Security solution -When user navigates to Add Elastic Rules Page -Then user can see available prebuilt rules to install -And user cannot Install those prebuilt rules -``` - -#### **Scenario: User with read privileges on security solution cannot update prebuilt rules** - -```Gherkin -Given a user with Security: read privileges on Security solution -When user navigates to Rule Updates Tab on Rules Management Page -Then user can see new updates for installed prebuilt rules -And user cannot Update those prebuilt rules -``` - -### Kibana upgrade - -#### **Scenario: User can operate with prebuilt rules when user upgrades from version `` to 8.9 version** - -```Gherkin -Given a user that is upgrading from version to version 8.9 -And the instance contains already installed prebuilt rules -When the upgrade is complete -Then user can install new prebuilt rules -And remove installed prebuilt rules -And update prebuilt rules from to 8.9 - -Examples: - | version | - | 8.7 | - | 7.17.x | -``` From 880f5f6905264288a833cf4e88de03b5d0108463 Mon Sep 17 00:00:00 2001 From: Georgii Gorbachev Date: Tue, 4 Jul 2023 20:42:24 +0200 Subject: [PATCH 13/13] Add test plan template --- .../cypress/test_plans/README.md | 13 ++- .../cypress/test_plans/test_plan_template.md | 101 ++++++++++++++++++ 2 files changed, 112 insertions(+), 2 deletions(-) create mode 100644 x-pack/plugins/security_solution/cypress/test_plans/test_plan_template.md diff --git a/x-pack/plugins/security_solution/cypress/test_plans/README.md b/x-pack/plugins/security_solution/cypress/test_plans/README.md index 6381b76017065..610c9a98947c9 100644 --- a/x-pack/plugins/security_solution/cypress/test_plans/README.md +++ b/x-pack/plugins/security_solution/cypress/test_plans/README.md @@ -2,7 +2,7 @@ This folder contains test plans for the features of Security Solution. -## Folder Structure +## Folder structure The folder is first split into major Security Solution domains: @@ -28,6 +28,15 @@ Within each subdomain, you can organize test plans as you like, for example: - you might want to have a folder per feature, if your features are large and you have multiple test plans per feature - or you might want to have a plain list of test plans if features are relatively small -## Ownership +## Folder ownership Each subdomain folder should be owned by a single GitHub team in the `.github/CODEOWNERS` file. + +## Test plan structure + +Some examples for reference: + +- [Test plan template](./test_plan_template.md). +- [Installation and Upgrade of Prebuilt Rules](./detection_response/prebuilt_rules/installation_and_upgrade.md). + +Feel free to tune the structure whenever it makes sense and improves readability or maintainability of your plan: add more sections to `Useful info`, add more top-level sections in addition to `Useful info` and `Scenarios`, etc. diff --git a/x-pack/plugins/security_solution/cypress/test_plans/test_plan_template.md b/x-pack/plugins/security_solution/cypress/test_plans/test_plan_template.md new file mode 100644 index 0000000000000..6f9a62ff4b612 --- /dev/null +++ b/x-pack/plugins/security_solution/cypress/test_plans/test_plan_template.md @@ -0,0 +1,101 @@ +# Awesome Feature + + +This is a test plan for ... + + +Status: `in progress`. + +## Useful information + +### Tickets + + + +- [Awesome Feature](https://github.com/elastic/security-team/issues/9999) epic +- [Add tests for the new awesome feature](https://github.com/elastic/kibana/issues/999999) +- [Document the new awesome feature](https://github.com/elastic/security-docs/issues/9999) + +### Terminology + + + +- **Term 1**: explanation. + +- **Term 2**: explanation. + +### Assumptions + + + +- Assumption 1. +- Assumption 2. + +### Non-functional requirements + + + +- Requirement 1. +- Requirement 2. + +## Scenarios + + + +### Section 1 + +#### **Scenario: Awesome feature works** + + +**Automation**: X e2e tests + Y integration tests + unit tests. + + +```Gherkin +Given ... +When ... +Then ... +``` + + + +### Section 2 + +#### **Scenario: ?** + +**Automation**: X e2e tests + Y integration tests + unit tests. + +```Gherkin +Given ... +When ... +Then ... +```