Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] JSON diffs test plan #175958

Merged
merged 2 commits into from
Feb 2, 2024
Merged

[Security Solution] JSON diffs test plan #175958

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f27642e
Update the test plan
nikitaindik Jan 31, 2024
63a30ac
Incorporate review feedback
nikitaindik Feb 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
219 changes: 218 additions & 1 deletion ...esting/test_plans/detection_response/prebuilt_rules/installation_and_upgrade.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,75 @@ This is a test plan for the workflows of installing and upgrading prebuilt rules

Status: `in progress`. The current test plan matches `Milestone 2` of the [Rule Immutability/Customization](https://github.com/elastic/security-team/issues/1974) epic. It does not cover any past functionality that was removed or functionality to be implemented in the future. The plan is about to change in the future Milestones.

## Table of Contents

- [Useful information](#useful-information)
- [Tickets](#tickets)
- [Terminology](#terminology)
- [Assumptions](#assumptions)
- [Non-functional requirements](#non-functional-requirements)
- [Functional requirements](#functional-requirements)
- [Scenarios](#scenarios)
- [Package installation](#package-installation)
- [**Scenario: Package is installed via Fleet**](#scenario-package-is-installed-via-fleet)
- [**Scenario: Package is installed via bundled Fleet package in Kibana**](#scenario-package-is-installed-via-bundled-fleet-package-in-kibana)
- [**Scenario: Large package can be installed on a small Kibana instance**](#scenario-large-package-can-be-installed-on-a-small-kibana-instance)
- [Rule installation and upgrade via the Prebuilt rules API](#rule-installation-and-upgrade-via-the-prebuilt-rules-api)
- [**Scenario: API can install all prebuilt rules**](#scenario-api-can-install-all-prebuilt-rules)
- [**Scenario: API can install prebuilt rules that are not yet installed**](#scenario-api-can-install-prebuilt-rules-that-are-not-yet-installed)
- [**Scenario: API can upgrade prebuilt rules that are outdated**](#scenario-api-can-upgrade-prebuilt-rules-that-are-outdated)
- [**Scenario: API does not install or upgrade prebuilt rules if they are up to date**](#scenario-api-does-not-install-or-upgrade-prebuilt-rules-if-they-are-up-to-date)
- [Scenarios for the real package](#scenarios-for-the-real-package)
- [**Scenario: User can install prebuilt rules from scratch, then install new rules and upgrade existing rules from the new package**](#scenario-user-can-install-prebuilt-rules-from-scratch-then-install-new-rules-and-upgrade-existing-rules-from-the-new-package)
- [Rule installation and upgrade notifications on the Rule Management page](#rule-installation-and-upgrade-notifications-on-the-rule-management-page)
- [**Scenario: User is NOT notified when no prebuilt rules are installed and there are no prebuilt rules assets**](#scenario-user-is-not-notified-when-no-prebuilt-rules-are-installed-and-there-are-no-prebuilt-rules-assets)
- [**Scenario: User is NOT notified when all prebuilt rules are installed and up to date**](#scenario-user-is-not-notified-when-all-prebuilt-rules-are-installed-and-up-to-date)
- [**Scenario: User is notified when no prebuilt rules are installed and there are rules available to install**](#scenario-user-is-notified-when-no-prebuilt-rules-are-installed-and-there-are-rules-available-to-install)
- [**Scenario: User is notified when some prebuilt rules can be installed**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-installed)
- [**Scenario: User is notified when some prebuilt rules can be upgraded**](#scenario-user-is-notified-when-some-prebuilt-rules-can-be-upgraded)
- [**Scenario: User is notified when both rules to install and upgrade are available**](#scenario-user-is-notified-when-both-rules-to-install-and-upgrade-are-available)
- [**Scenario: User is notified after a prebuilt rule gets deleted**](#scenario-user-is-notified-after-a-prebuilt-rule-gets-deleted)
- [Rule installation workflow: base cases](#rule-installation-workflow-base-cases)
- [**Scenario: User can install prebuilt rules one by one**](#scenario-user-can-install-prebuilt-rules-one-by-one)
- [**Scenario: User can install multiple prebuilt rules selected on the page**](#scenario-user-can-install-multiple-prebuilt-rules-selected-on-the-page)
- [**Scenario: User can install all available prebuilt rules at once**](#scenario-user-can-install-all-available-prebuilt-rules-at-once)
- [**Scenario: Empty screen is shown when all prebuilt rules are installed**](#scenario-empty-screen-is-shown-when-all-prebuilt-rules-are-installed)
- [**Scenario: User can preview rules available for installation**](#scenario-user-can-preview-rules-available-for-installation)
- [**Scenario: User can install a rule using the rule preview**](#scenario-user-can-install-a-rule-using-the-rule-preview)
- [**Scenario: User can see correct rule information in preview before installing**](#scenario-user-can-see-correct-rule-information-in-preview-before-installing)
- [**Scenario: Tabs and sections without content should be hidden in preview before installing**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-installing)
- [Rule installation workflow: filtering, sorting, pagination](#rule-installation-workflow-filtering-sorting-pagination)
- [Rule installation workflow: misc cases](#rule-installation-workflow-misc-cases)
- [**Scenario: User opening the Add Rules page sees a loading skeleton until the package installation is completed**](#scenario-user-opening-the-add-rules-page-sees-a-loading-skeleton-until-the-package-installation-is-completed)
- [**Scenario: User can navigate from the Add Rules page to the Rule Management page via breadcrumbs**](#scenario-user-can-navigate-from-the-add-rules-page-to-the-rule-management-page-via-breadcrumbs)
- [Rule upgrade workflow: base cases](#rule-upgrade-workflow-base-cases)
- [**Scenario: User can upgrade prebuilt rules one by one**](#scenario-user-can-upgrade-prebuilt-rules-one-by-one)
- [**Scenario: User can upgrade multiple prebuilt rules selected on the page**](#scenario-user-can-upgrade-multiple-prebuilt-rules-selected-on-the-page)
- [**Scenario: User can upgrade all available prebuilt rules at once**](#scenario-user-can-upgrade-all-available-prebuilt-rules-at-once)
- [**Scenario: User can preview rules available for upgrade**](#scenario-user-can-preview-rules-available-for-upgrade)
- [**Scenario: User can upgrade a rule using the rule preview**](#scenario-user-can-upgrade-a-rule-using-the-rule-preview)
- [**Scenario: User can see correct rule information in preview before upgrading**](#scenario-user-can-see-correct-rule-information-in-preview-before-upgrading)
- [**Scenario: Tabs and sections without content should be hidden in preview before upgrading**](#scenario-tabs-and-sections-without-content-should-be-hidden-in-preview-before-upgrading)
- [Rule upgrade workflow: filtering, sorting, pagination](#rule-upgrade-workflow-filtering-sorting-pagination)
- [Rule upgrade workflow: viewing rule changes in JSON diff view](#rule-upgrade-workflow-viewing-rule-changes-in-json-diff-view)
- [**Scenario: User can see changes in a side-by-side JSON diff view**](#scenario-user-can-see-changes-in-a-side-by-side-json-diff-view)
- [**Scenario: User can see precisely how property values would change after upgrade**](#scenario-user-can-see-precisely-how-property-values-would-change-after-upgrade)
- [**Scenario: Rule actions and exception lists should not be shown as modified**](#scenario-rule-actions-and-exception-lists-should-not-be-shown-as-modified)
- [**Scenario: Dynamic properties should not be included in preview**](#scenario-dynamic-properties-should-not-be-included-in-preview)
- [**Scenario: Technical properties should not be included in preview**](#scenario-technical-properties-should-not-be-included-in-preview)
- [**Scenario: Properties with semantically equal values should not be shown as modified**](#scenario-properties-with-semantically-equal-values-should-not-be-shown-as-modified)
- [**Scenario: Unchanged sections of a rule should be hidden by default**](#scenario-unchanged-sections-of-a-rule-should-be-hidden-by-default)
- [**Scenario: Properties should be sorted alphabetically**](#scenario-properties-should-be-sorted-alphabetically)
- [Rule upgrade workflow: misc cases](#rule-upgrade-workflow-misc-cases)
- [**Scenario: User doesn't see the Rule Updates tab until the package installation is completed**](#scenario-user-doesnt-see-the-rule-updates-tab-until-the-package-installation-is-completed)
- [Error handling](#error-handling)
- [**Scenario: Error is handled when any operation on prebuilt rules fails**](#scenario-error-is-handled-when-any-operation-on-prebuilt-rules-fails)
- [Authorization / RBAC](#authorization--rbac)
- [**Scenario: User with read privileges on Security Solution cannot install prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-install-prebuilt-rules)
- [**Scenario: User with read privileges on Security Solution cannot upgrade prebuilt rules**](#scenario-user-with-read-privileges-on-security-solution-cannot-upgrade-prebuilt-rules)
- [Kibana upgrade](#kibana-upgrade)
- [**Scenario: User can use prebuilt rules after upgrading Kibana from version A to B**](#scenario-user-can-use-prebuilt-rules-after-upgrading-kibana-from-version-a-to-b)

## Useful information

### Tickets
Expand Down Expand Up @@ -704,7 +773,9 @@ When user opens the Rule Updates table
Then all X rules available for upgrade should be displayed in the table
When user opens the rule preview for the 1st rule
Then the preview should open
And all properties of the new version of the 1st rule should be displayed in the correct tab and section of the preview (see examples of rule properties above)
And the "Updates" tab should be active
When user selects the "Overview" tab
Then all properties of the new version of the 1st rule should be displayed in the correct tab and section of the preview (see examples of rule properties above)
When user selects the 2nd rule in the table
Then the preview should be updated
And all properties of the new version of the 2nd rule should be displayed in the correct tab and section of the preview (see examples of rule properties above)
Expand Down Expand Up @@ -732,6 +803,152 @@ And the Investigation Guide tab should NOT be displayed

TODO: add scenarios https://github.com/elastic/kibana/issues/166215

### Rule upgrade workflow: viewing rule changes in JSON diff view

#### **Scenario: User can see changes in a side-by-side JSON diff view**

**Automation**: 1 e2e test

```Gherkin
Given X prebuilt rules are installed in Kibana
And for Y of these rules new versions are available
When user opens the Rule Updates table and selects a rule
Then the upgrade preview should open
And rule changes should be displayed in a two-column JSON diff view
And correct rule version numbers should be displayed in their respective columns
When the user selects another rule without closing the preview
Then the preview should display the changes for the newly selected rule
```

#### **Scenario: User can see precisely how property values would change after upgrade**

**Automation**: 1 UI integration test

```Gherkin
Given a rule preview with rule changes is open
Then each line of <column> that was <change_type> should have <bg_color> background
And marked with <line_badge> badge
And each changed word in <column> should be highlighted with <accent_color>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot recall now, but we are only using only WORD diffing for the whole JSON, right? No need to test other uses cases, like character diffing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, at this stage it's by word only.


Examples:
| change_type | column | bg_color | accent_color | line_badge |
| updated | Current rule | removed_bg_color | removed_accent_color | - |
| updated | Elastic update | added_bg_color | added_accent_color | + |
| removed | Current rule | removed_bg_color | none | - |
| removed | Elastic update | none | none | none |
| added | Current rule | none | none | none |
| added | Elastic update | added_bg_color | none | + |
```

#### **Scenario: Rule actions and exception lists should not be shown as modified**

**Automation**: 1 UI integration test

```Gherkin
Given a prebuilt rule is installed in Kibana
And the currently installed version of this rule doesn't have any actions or an exception list
And a user has set up actions and an exception list for this rule
And this rule has an update available
And the update doesn't define any actions or an exception list
When a user opens the upgrade preview for this rule
Then the preview should open
And the JSON diff shouldn't show any modifications to rule's actions or exception list
```

#### **Scenario: Dynamic properties should not be included in preview**

**Automation**: 1 e2e test

```Gherkin
Given a prebuilt rule is installed in Kibana
And this rule is disabled by default
And a user has enabled this rule
And this rule executed at least once
And this rule has an update available
When user opens the upgrade preview
Then the preview should open
And the JSON diff shouldn't show any <property> properties on both sides

Examples:
| property |
| execution_summary |
| enabled |
```

#### **Scenario: Technical properties should not be included in preview**

**Automation**: 1 UI integration test

```Gherkin
Given a prebuilt rule is installed in Kibana
And this rule has an update available
When a user opens the upgrade preview
Then the preview should open
And the JSON diff shouldn't show any <technical_property> properties on both sides

Examples:
| technical_property |
| revision |
| updated_at |
| updated_by |
| created_at |
| created_by |
```

#### **Scenario: Properties with semantically equal values should not be shown as modified**

**Automation**: 1 UI integration test

```Gherkin
Given a prebuilt rule is installed in Kibana
And this rule has an update available
And the update has properties with different, but semantically equal values
When a user opens the upgrade preview
Then the preview should open
And the JSON diff shouldn't show any changes to properties with semantically equal values

Duration examples:
| 1h |
| 60m |
| 3600s |

Empty value examples:
| no value |
| '' |
| [] |
| undefined |
| null |
```

#### **Scenario: Unchanged sections of a rule should be hidden by default**

**Automation**: 1 UI integration test

```Gherkin
Given a prebuilt rule is installed in Kibana
And this rule has an update available
When a user opens the upgrade preview
Then the preview should open
And only the sections of the diff that have changes should be visible
And unchanged sections should be hidden behind a button with a number of unchanged lines
When a user clicks on the hidden section button
Then the section should expand and show the unchanged properties
```

#### **Scenario: Properties should be sorted alphabetically**

**Automation**: 1 UI integration test

```Gherkin
Given a prebuilt rule is installed in Kibana
And this rule has an update available
When a user opens the upgrade preview
Then the preview should open
And visible properties should be sorted alphabetically
When a user expands all hidden sections
Then all properties of the rule should be sorted alphabetically
nikitaindik marked this conversation as resolved.
Show resolved Hide resolved
```

### Rule upgrade workflow: misc cases

#### **Scenario: User doesn't see the Rule Updates tab until the package installation is completed**
Expand Down