Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FR] [Implementation] Smart Limits for Detection Rules #4388

Open
shashank-elastic opened this issue Jan 16, 2025 · 7 comments · May be fixed by #4419
Open

[FR] [Implementation] Smart Limits for Detection Rules #4388

shashank-elastic opened this issue Jan 16, 2025 · 7 comments · May be fixed by #4419
Assignees
@shashank-elastic
Labels
community enhancement New feature or request Team: TRADE

Comments

@shashank-elastic
Copy link
Contributor

Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

Per discussion(s) in - #4150, the previous limit for rule assets was removed to introduce historical package versions in 8.17.1 and 8.17.2. With this change the current asset count in rule package is close ~11500 and the tested asset limit on serverless projects is ~15000 before we hit OOM issues on serverless instances.

We need to reintroduce "smart limits" on detection rules to ensure there is minimum disruption to rule version diff feature in product that is used to analyse rule upgrades.

Desired Solution

With some brainstorming ides on the SWG call, there are 2 possible solutions to this issue.

Solution 1

  • Reintroduce limits to hold have at most 5 versions of each rule
  • More than 5 versions, start removing the oldest version of the rule.

Solution 2

  • When the total assets increase 15000 rule assets, removing the excess assets and bring it back to the desired tested number
  • When package is built if we have 15010 rules, then the excess 10 rule assets which are of the oldest nature will be removed
    • With this we could potentially remove more than 1 version of the same rule becuase its the oldest in nature
    • We should ensure latest versions of the rule even if its oldest is retained.

For both of these solutions we need to revist code at keep_latest_versions

Considered Alternatives

Be able to handle rule asset management from the product side, this is being considered as a solution to ensure there is enough time for product enhancements

Additional Context

#4150
@shashank-elastic shashank-elastic added enhancement New feature or request Team: TRADE labels Jan 16, 2025
@shashank-elastic shashank-elastic self-assigned this Jan 16, 2025
@shashank-elastic shashank-elastic mentioned this issue Jan 16, 2025
Closed
@shashank-elastic
Copy link
Contributor Author

Pseudo Code and Preliminary Testing

  • Currently we have around 11578 assets, so testing we are assuming the limits to be 11500
print(f"Total number of rules: {len(assets)}")
        if len(assets) > 11500:
            # Flatten the list of all versions and sort by version (older first)
            all_versions = sorted(
                [(base_id, version, key) for base_id, versions in rule_versions.items() for version, key in versions],
                key=lambda x: x[1]
            )

            # Calculate the number of excess assets
            excess_count = len(assets) - 11500
            print(f"Excess count: {excess_count}")
            processed_base_ids = set()
            # Remove the oldest excess assets
            for base_id, version, key in all_versions:
                if base_id not in processed_base_ids and key in assets:
                    del assets[key]
                    processed_base_ids.add(base_id)
                    print(f"Removed rule {key} with version {version}")
                    excess_count -= 1
                if excess_count <= 0:
                    break
  • Some verbose testing and output
  • 78 Assets identified as excess which are rightly considered for removal
  • The order of assets removal need to be tested we need to consider oldest asset first, and then remove its older version!
  •    all_versions = sorted(
          [(base_id, version, key) for base_id, versions in rule_versions.items() for version, key in versions],
          key=lambda x: x[1], reverse = True
      )
  • This will fetch older assets, and then proceed further.
❯ python -m detection_rules dev build-release              
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building package 8.18
 - 4 rules excluded from package
Rule changes detected!
 - 951 changed rules
 - 1 new rules
 - 0 newly deprecated rules
run `build-release --update-version-lock` to update version.lock.json and deprecated_rules.json
Package saved to: /Users/shashankks/elastic_workspace/detection-rules/releases/8.18
loaded security_detection_engine manifests from the following package versions: ['8.17.3', '8.17.2', '8.17.1', '8.16.5', '8.16.4', '8.16.3', '8.16.2', '8.16.1', '8.15.14', '8.15.13', '8.15.12', '8.15.11', '8.15.10', '8.15.9', '8.15.8', '8.15.7', '8.15.6', '8.15.5', '8.15.4', '8.15.3', '8.15.2', '8.15.1', '8.14.20', '8.14.19', '8.14.18', '8.14.17', '8.14.16', '8.14.15', '8.14.14', '8.14.13', '8.14.12', '8.14.11', '8.14.10', '8.14.9', '8.14.8', '8.14.7', '8.14.6', '8.14.5', '8.14.4', '8.14.3', '8.14.2', '8.14.1', '8.13.23', '8.13.22', '8.13.21', '8.13.20', '8.13.19', '8.13.18', '8.13.17', '8.13.16', '8.13.15', '8.13.14', '8.13.13', '8.13.12', '8.13.11', '8.13.10', '8.13.9', '8.13.8', '8.13.7', '8.13.6', '8.13.5', '8.13.4', '8.13.3', '8.13.2', '8.13.1', '8.12.26', '8.12.25', '8.12.24', '8.12.23', '8.12.22', '8.12.21', '8.12.20', '8.12.19', '8.12.18', '8.12.17', '8.12.16', '8.12.15', '8.12.14', '8.12.13', '8.12.12', '8.12.11', '8.12.10', '8.12.9', '8.12.8', '8.12.7', '8.12.6', '8.12.5', '8.12.4', '8.12.3', '8.12.2', '8.12.1', '8.11.21', '8.11.20', '8.11.19', '8.11.18', '8.11.17', '8.11.16', '8.11.15', '8.11.14', '8.11.13', '8.11.12', '8.11.11', '8.11.10', '8.11.9', '8.11.8', '8.11.7', '8.11.6', '8.11.5', '8.11.4', '8.11.3', '8.11.2', '8.11.1', '8.10.18', '8.10.17', '8.10.16', '8.10.15', '8.10.14', '8.10.13', '8.10.12', '8.10.11', '8.10.10', '8.10.9', '8.10.8', '8.10.7', '8.10.6', '8.10.5', '8.10.4', '8.10.3', '8.10.2', '8.10.1', '8.9.15', '8.9.14', '8.9.13', '8.9.12', '8.9.11', '8.9.10', '8.9.9', '8.9.8', '8.9.7', '8.9.6', '8.9.5', '8.9.4', '8.9.3', '8.9.2', '8.9.1', '8.8.15', '8.8.14', '8.8.13', '8.8.12', '8.8.11', '8.8.10', '8.8.9', '8.8.8', '8.8.7', '8.8.6', '8.8.5', '8.8.4', '8.8.3', '8.8.2', '8.8.1', '8.7.13', '8.7.12', '8.7.11', '8.7.10', '8.7.9', '8.7.8', '8.7.7', '8.7.6', '8.7.5', '8.7.4', '8.7.3', '8.7.2', '8.7.1', '8.6.10', '8.6.9', '8.6.8', '8.6.7', '8.6.6', '8.6.5', '8.6.4', '8.6.3', '8.6.2', '8.6.1', '8.5.8', '8.5.7', '8.5.6', '8.5.5', '8.5.4', '8.5.3', '8.5.2', '8.5.1', '8.4.5', '8.4.4', '8.4.3', '8.4.2', '8.4.1', '8.3.4', '8.3.3', '8.3.2', '8.3.1', '8.2.1', '8.1.1', '1.0.2', '1.0.1']
[+] Adding all historical rule versions in our release package for version             8.18.0
Total number of rules: 11578
Excess count: 78
Removed rule 0049cf71-fe13-4d79-b767-f7519921ffb5_1 with version 1
Removed rule 00678712-b2df-11ed-afe9-f661ea17fbcc_1 with version 1
Removed rule 0136b315-b566-482f-866c-1d8e2477ba16_1 with version 1
Removed rule 015cca13-8832-49ac-a01b-a396114809f6_1 with version 1
Removed rule 0171f283-ade7-4f87-9521-ac346c68cc9b_1 with version 1
Removed rule 017de1e4-ea35-11ee-a417-f661ea17fbce_1 with version 1
Removed rule 01c49712-25bc-49d2-a27d-d7ce52f5dc49_1 with version 1
Removed rule 027ff9ea-85e7-42e3-99d2-bbb7069e02eb_1 with version 1
Removed rule 0294f105-d7af-4a02-ae90-35f56763ffa2_1 with version 1
Removed rule 02a23ee7-c8f8-4701-b99d-e9038ce313cb_1 with version 1
Removed rule 02bab13d-fb14-4d7c-b6fe-4a28874d37c5_1 with version 1
Removed rule 02ea4563-ec10-4974-b7de-12e65aa4f9b3_1 with version 1
Removed rule 035a6f21-4092-471d-9cda-9e379f459b1e_1 with version 1
Removed rule 0369e8a6-0fa7-4e7a-961a-53180a4c966e_1 with version 1
Removed rule 03a514d9-500e-443e-b6a9-72718c548f6c_1 with version 1
Removed rule 03c23d45-d3cb-4ad4-ab5d-b361ffe8724a_1 with version 1
Removed rule 0415258b-a7b2-48a6-891a-3367cd9d4d31_1 with version 1
Removed rule 0415f22a-2336-45fa-ba07-618a5942e22c_1 with version 1
Removed rule 043d80a3-c49e-43ef-9c72-1088f0c7b278_1 with version 1
Removed rule 04e65517-16e9-4fc4-b7f1-94dc21ecea0d_1 with version 1
Removed rule 054db96b-fd34-43b3-9af2-587b3bd33964_1 with version 1
Removed rule 05cad2fb-200c-407f-b472-02ea8c9e5e4a_1 with version 1
Removed rule 06568a02-af29-4f20-929c-f3af281e41aa_1 with version 1
Removed rule 0678bc9c-b71a-433b-87e6-2f664b6b3131_1 with version 1
Removed rule 06a7a03c-c735-47a6-a313-51c354aef6c3_1 with version 1
Removed rule 06d555e4-c8ce-4d90-90e1-ec7f66df5a6a_1 with version 1
Removed rule 06f3a26c-ea35-11ee-a417-f661ea17fbce_1 with version 1
Removed rule 07639887-da3a-4fbf-9532-8ce748ff8c50_1 with version 1
Removed rule 0787daa6-f8c5-453b-a4ec-048037f6c1cd_1 with version 1
Removed rule 07b1ef73-1fde-4a49-a34a-5dd40011b076_1 with version 1
Removed rule 080bc66a-5d56-4d1f-8071-817671716db9_1 with version 1
Removed rule 083fa162-e790-4d85-9aeb-4fea04188adb_1 with version 1
Removed rule 0859355c-0f08-4b43-8ff5-7d2a4789fc08_1 with version 1
Removed rule 089db1af-740d-4d84-9a5b-babd6de143b0_1 with version 1
Removed rule 092b068f-84ac-485d-8a55-7dd9e006715f_1 with version 1
Removed rule 095b6a58-8f88-4b59-827c-ab584ad4e759_1 with version 1
Removed rule 09bc6c90-7501-494d-b015-5d988dc3f233_1 with version 1
Removed rule 09d028a5-dcde-409f-8ae0-557cef1b7082_1 with version 1
Removed rule 0ab319ef-92b8-4c7f-989b-5de93c852e93_1 with version 1
Removed rule 0abf0c5b-62dd-48d2-ac4e-6b43fe3a6e83_1 with version 1
Removed rule 0b15bcad-aff1-4250-a5be-5d1b7eb56d07_1 with version 1
Removed rule 0b2f3da5-b5ec-47d1-908b-6ebb74814289_1 with version 1
Removed rule 0b76ad27-c3f3-4769-9e7e-3237137fdf06_1 with version 1
Removed rule 0b79f5c0-2c31-4fea-86cd-e62644278205_1 with version 1
Removed rule 0b803267-74c5-444d-ae29-32b5db2d562a_1 with version 1
Removed rule 0b96dfd8-5b8c-4485-9a1c-69ff7839786a_1 with version 1
Removed rule 0c093569-dff9-42b6-87b1-0242d9f7d9b4_1 with version 1
Removed rule 0c1e8fda-4f09-451e-bc77-a192b6cbfc32_1 with version 1
Removed rule 0c41e478-5263-4c69-8f9e-7dfd2c22da64_1 with version 1
Removed rule 0c74cd7e-ea35-11ee-a417-f661ea17fbce_1 with version 1
Removed rule 0cd2f3e6-41da-40e6-b28b-466f688f00a6_1 with version 1
Removed rule 0ce6487d-8069-4888-9ddd-61b52490cebc_1 with version 1
Removed rule 0d160033-fab7-4e72-85a3-3a9d80c8bff7_1 with version 1
Removed rule 0e1af929-42ed-4262-a846-55a7c54e7c84_1 with version 1
Removed rule 0e4367a0-a483-439d-ad2e-d90500b925fd_1 with version 1
Removed rule 0ef5d3eb-67ef-43ab-93b7-305cfa5a21f6_1 with version 1
Removed rule 0f4d35e4-925e-4959-ab24-911be207ee6f_1 with version 1
Removed rule 0f56369f-eb3d-459c-a00b-87c2bf7bdfc5_1 with version 1
Removed rule 0f615fe4-eaa2-11ee-ae33-f661ea17fbce_1 with version 1
Removed rule 0f93cb9a-1931-48c2-8cd0-f173fd3e5283_1 with version 1
Removed rule 0ff84c42-873d-41a2-a4ed-08d74d352d01_1 with version 1
Removed rule 10445cf0-0748-11ef-ba75-f661ea17fbcc_1 with version 1
Removed rule 10f3d520-ea35-11ee-a417-f661ea17fbce_1 with version 1
Removed rule 119c8877-8613-416d-a98a-96b6664ee73a_1 with version 1
Removed rule 11dd9713-0ec6-4110-9707-32daae1ee68c_1 with version 1
Removed rule 12051077-0124-4394-9522-8f4f4db1d674_1 with version 1
Removed rule 1224da6c-0326-4b4f-8454-68cdc5ae542b_1 with version 1
Removed rule 1251b98a-ff45-11ee-89a1-f661ea17fbce_1 with version 1
Removed rule 128468bf-cab1-4637-99ea-fdf3780a4609_1 with version 1
Removed rule 12de29d4-bbb0-4eef-b687-857e8a163870_1 with version 1
Removed rule 135abb91-dcf4-48aa-b81a-5ad036b67c68_1 with version 1
Removed rule 1397e1b9-0c90-4d24-8d7b-80598eb9bc9a_1 with version 1
Removed rule 13e908b9-7bf0-4235-abc9-b5deb500d0ad_1 with version 1
Removed rule 14dab405-5dd9-450c-8106-72951af2391f_1 with version 1
Removed rule 14ed1aa9-ebfd-4cf9-a463-0ac59ec55204_1 with version 1
Removed rule 1502a836-84b2-11ef-b026-f661ea17fbcc_1 with version 1
Removed rule 151d8f72-0747-11ef-a0c2-f661ea17fbcc_1 with version 1
Removed rule 1542fa53-955e-4330-8e4d-b2d812adeb5f_1 with version 1
(.venv) 
detection-rules on  main [$!?] is 📦 v0.4.3 via 🐍 v3.12.8 (.venv) on ☁️  [email protected] took 1m23s 
❯

@Mikaayenson
Copy link
Contributor

what if there is only one version?

@shashank-elastic
Copy link
Contributor Author

shashank-elastic commented Jan 24, 2025 •
edited
Loading

Update 24 Jan Pseudo Code and Preliminary Testing

  • We introduced count of versions when flattening the oldest versions, to ensure base rule with 1 version is not removed
  • This also gave us the flexibility of knowing how many versions each oldest based_id has
filtered_assets = {}
        print(f"Total number of rules: {len(assets)}")
        if len(assets) > 11500:
            # Flatten the list of all versions and sort by version (older first)
            all_versions = sorted(
                [(base_id, version, key, len(versions)) for base_id, versions in rule_versions.items() for version, key in versions],
                key=lambda x: x[0]
            )


            # Calculate the number of excess assets
            excess_count = len(assets) - 11500
            print(f"Excess count: {excess_count}")
            processed_base_ids = set()
            # Remove the oldest excess assets
            for base_id, versions, key, count in all_versions:
                if base_id not in processed_base_ids and key in assets:
                   # for method 2 we could make count >6
                    if count > 1: 
                        # Rule has more than one version can be removed 
                        print(f"Removed asset {key} of rule {base_id} and version {versions} which had total {count} versions")
                        del assets[key]
                    processed_base_ids.add(base_id)
                    excess_count -= 1
                if excess_count <= 0:
                    break

❯ python -m detection_rules dev build-release 
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building package 8.18
 - 4 rules excluded from package
Rule changes detected!
 - 951 changed rules
 - 1 new rules
 - 0 newly deprecated rules
run `build-release --update-version-lock` to update version.lock.json and deprecated_rules.json
Package saved to: /Users/shashankks/elastic_workspace/detection-rules/releases/8.18
loaded security_detection_engine manifests from the following package versions: ['8.17.3', '8.17.2', '8.17.1', '8.16.5', '8.16.4', '8.16.3', '8.16.2', '8.16.1', '8.15.14', '8.15.13', '8.15.12', '8.15.11', '8.15.10', '8.15.9', '8.15.8', '8.15.7', '8.15.6', '8.15.5', '8.15.4', '8.15.3', '8.15.2', '8.15.1', '8.14.20', '8.14.19', '8.14.18', '8.14.17', '8.14.16', '8.14.15', '8.14.14', '8.14.13', '8.14.12', '8.14.11', '8.14.10', '8.14.9', '8.14.8', '8.14.7', '8.14.6', '8.14.5', '8.14.4', '8.14.3', '8.14.2', '8.14.1', '8.13.23', '8.13.22', '8.13.21', '8.13.20', '8.13.19', '8.13.18', '8.13.17', '8.13.16', '8.13.15', '8.13.14', '8.13.13', '8.13.12', '8.13.11', '8.13.10', '8.13.9', '8.13.8', '8.13.7', '8.13.6', '8.13.5', '8.13.4', '8.13.3', '8.13.2', '8.13.1', '8.12.26', '8.12.25', '8.12.24', '8.12.23', '8.12.22', '8.12.21', '8.12.20', '8.12.19', '8.12.18', '8.12.17', '8.12.16', '8.12.15', '8.12.14', '8.12.13', '8.12.12', '8.12.11', '8.12.10', '8.12.9', '8.12.8', '8.12.7', '8.12.6', '8.12.5', '8.12.4', '8.12.3', '8.12.2', '8.12.1', '8.11.21', '8.11.20', '8.11.19', '8.11.18', '8.11.17', '8.11.16', '8.11.15', '8.11.14', '8.11.13', '8.11.12', '8.11.11', '8.11.10', '8.11.9', '8.11.8', '8.11.7', '8.11.6', '8.11.5', '8.11.4', '8.11.3', '8.11.2', '8.11.1', '8.10.18', '8.10.17', '8.10.16', '8.10.15', '8.10.14', '8.10.13', '8.10.12', '8.10.11', '8.10.10', '8.10.9', '8.10.8', '8.10.7', '8.10.6', '8.10.5', '8.10.4', '8.10.3', '8.10.2', '8.10.1', '8.9.15', '8.9.14', '8.9.13', '8.9.12', '8.9.11', '8.9.10', '8.9.9', '8.9.8', '8.9.7', '8.9.6', '8.9.5', '8.9.4', '8.9.3', '8.9.2', '8.9.1', '8.8.15', '8.8.14', '8.8.13', '8.8.12', '8.8.11', '8.8.10', '8.8.9', '8.8.8', '8.8.7', '8.8.6', '8.8.5', '8.8.4', '8.8.3', '8.8.2', '8.8.1', '8.7.13', '8.7.12', '8.7.11', '8.7.10', '8.7.9', '8.7.8', '8.7.7', '8.7.6', '8.7.5', '8.7.4', '8.7.3', '8.7.2', '8.7.1', '8.6.10', '8.6.9', '8.6.8', '8.6.7', '8.6.6', '8.6.5', '8.6.4', '8.6.3', '8.6.2', '8.6.1', '8.5.8', '8.5.7', '8.5.6', '8.5.5', '8.5.4', '8.5.3', '8.5.2', '8.5.1', '8.4.5', '8.4.4', '8.4.3', '8.4.2', '8.4.1', '8.3.4', '8.3.3', '8.3.2', '8.3.1', '8.2.1', '8.1.1', '1.0.2', '1.0.1']
[+] Adding all historical rule versions in our release package for version             8.18.0
Total number of rules: 11578
Excess count: 78
Removed asset 000047bb-b27a-47ec-8b62-ef1a5d2c9e19_100 of rule 000047bb-b27a-47ec-8b62-ef1a5d2c9e19 and version 100 which had total 17 versions
Removed asset 00140285-b827-4aee-aa09-8113f58a08f3_100 of rule 00140285-b827-4aee-aa09-8113f58a08f3 and version 100 which had total 23 versions
Removed asset 0022d47d-39c7-4f69-a232-4fe9dc7a3acd_10 of rule 0022d47d-39c7-4f69-a232-4fe9dc7a3acd and version 10 which had total 22 versions
Removed asset 00678712-b2df-11ed-afe9-f661ea17fbcc_1 of rule 00678712-b2df-11ed-afe9-f661ea17fbcc and version 1 which had total 3 versions
Removed asset 0136b315-b566-482f-866c-1d8e2477ba16_1 of rule 0136b315-b566-482f-866c-1d8e2477ba16 and version 1 which had total 9 versions
Removed asset 015cca13-8832-49ac-a01b-a396114809f6_1 of rule 015cca13-8832-49ac-a01b-a396114809f6 and version 1 which had total 10 versions
Removed asset 0171f283-ade7-4f87-9521-ac346c68cc9b_1 of rule 0171f283-ade7-4f87-9521-ac346c68cc9b and version 1 which had total 7 versions
Removed asset 01c49712-25bc-49d2-a27d-d7ce52f5dc49_1 of rule 01c49712-25bc-49d2-a27d-d7ce52f5dc49 and version 1 which had total 4 versions
Removed asset 027ff9ea-85e7-42e3-99d2-bbb7069e02eb_1 of rule 027ff9ea-85e7-42e3-99d2-bbb7069e02eb and version 1 which had total 12 versions
Removed asset 0294f105-d7af-4a02-ae90-35f56763ffa2_1 of rule 0294f105-d7af-4a02-ae90-35f56763ffa2 and version 1 which had total 4 versions
Removed asset 02a23ee7-c8f8-4701-b99d-e9038ce313cb_1 of rule 02a23ee7-c8f8-4701-b99d-e9038ce313cb and version 1 which had total 6 versions
Removed asset 02a4576a-7480-4284-9327-548a806b5e48_100 of rule 02a4576a-7480-4284-9327-548a806b5e48 and version 100 which had total 14 versions
Removed asset 02bab13d-fb14-4d7c-b6fe-4a28874d37c5_1 of rule 02bab13d-fb14-4d7c-b6fe-4a28874d37c5 and version 1 which had total 3 versions
Removed asset 02ea4563-ec10-4974-b7de-12e65aa4f9b3_1 of rule 02ea4563-ec10-4974-b7de-12e65aa4f9b3 and version 1 which had total 9 versions
Removed asset 03024bd9-d23f-4ec1-8674-3cf1a21e130b_100 of rule 03024bd9-d23f-4ec1-8674-3cf1a21e130b and version 100 which had total 9 versions
Removed asset 035889c4-2686-4583-a7df-67f89c292f2c_100 of rule 035889c4-2686-4583-a7df-67f89c292f2c and version 100 which had total 18 versions
Removed asset 035a6f21-4092-471d-9cda-9e379f459b1e_1 of rule 035a6f21-4092-471d-9cda-9e379f459b1e and version 1 which had total 3 versions
Removed asset 0369e8a6-0fa7-4e7a-961a-53180a4c966e_1 of rule 0369e8a6-0fa7-4e7a-961a-53180a4c966e and version 1 which had total 3 versions
Removed asset 03a514d9-500e-443e-b6a9-72718c548f6c_1 of rule 03a514d9-500e-443e-b6a9-72718c548f6c and version 1 which had total 2 versions
Removed asset 03c23d45-d3cb-4ad4-ab5d-b361ffe8724a_1 of rule 03c23d45-d3cb-4ad4-ab5d-b361ffe8724a and version 1 which had total 3 versions
Removed asset 0415f22a-2336-45fa-ba07-618a5942e22c_1 of rule 0415f22a-2336-45fa-ba07-618a5942e22c and version 1 which had total 13 versions
Removed asset 043d80a3-c49e-43ef-9c72-1088f0c7b278_1 of rule 043d80a3-c49e-43ef-9c72-1088f0c7b278 and version 1 which had total 6 versions
Removed asset 04c5a96f-19c5-44fd-9571-a0b033f9086f_100 of rule 04c5a96f-19c5-44fd-9571-a0b033f9086f and version 100 which had total 5 versions
Removed asset 053a0387-f3b5-4ba5-8245-8002cca2bd08_100 of rule 053a0387-f3b5-4ba5-8245-8002cca2bd08 and version 100 which had total 16 versions
Removed asset 054db96b-fd34-43b3-9af2-587b3bd33964_1 of rule 054db96b-fd34-43b3-9af2-587b3bd33964 and version 1 which had total 7 versions
Removed asset 0564fb9d-90b9-4234-a411-82a546dc1343_100 of rule 0564fb9d-90b9-4234-a411-82a546dc1343 and version 100 which had total 19 versions
Removed asset 05b358de-aa6d-4f6c-89e6-78f74018b43b_100 of rule 05b358de-aa6d-4f6c-89e6-78f74018b43b and version 100 which had total 17 versions
Removed asset 05cad2fb-200c-407f-b472-02ea8c9e5e4a_1 of rule 05cad2fb-200c-407f-b472-02ea8c9e5e4a and version 1 which had total 4 versions
Removed asset 05e5a668-7b51-4a67-93ab-e9af405c9ef3_100 of rule 05e5a668-7b51-4a67-93ab-e9af405c9ef3 and version 100 which had total 11 versions
Removed asset 0635c542-1b96-4335-9b47-126582d2c19a_100 of rule 0635c542-1b96-4335-9b47-126582d2c19a and version 100 which had total 20 versions
Removed asset 06568a02-af29-4f20-929c-f3af281e41aa_1 of rule 06568a02-af29-4f20-929c-f3af281e41aa and version 1 which had total 11 versions
Removed asset 0678bc9c-b71a-433b-87e6-2f664b6b3131_1 of rule 0678bc9c-b71a-433b-87e6-2f664b6b3131 and version 1 which had total 4 versions
Removed asset 06a7a03c-c735-47a6-a313-51c354aef6c3_1 of rule 06a7a03c-c735-47a6-a313-51c354aef6c3 and version 1 which had total 12 versions
Removed asset 06d555e4-c8ce-4d90-90e1-ec7f66df5a6a_1 of rule 06d555e4-c8ce-4d90-90e1-ec7f66df5a6a and version 1 which had total 2 versions
Removed asset 06dceabf-adca-48af-ac79-ffdf4c3b1e9a_100 of rule 06dceabf-adca-48af-ac79-ffdf4c3b1e9a and version 100 which had total 19 versions
Removed asset 074464f9-f30d-4029-8c03-0ed237fffec7_100 of rule 074464f9-f30d-4029-8c03-0ed237fffec7 and version 100 which had total 19 versions
Removed asset 07639887-da3a-4fbf-9532-8ce748ff8c50_1 of rule 07639887-da3a-4fbf-9532-8ce748ff8c50 and version 1 which had total 6 versions
Removed asset 0787daa6-f8c5-453b-a4ec-048037f6c1cd_1 of rule 0787daa6-f8c5-453b-a4ec-048037f6c1cd and version 1 which had total 7 versions
Removed asset 07b1ef73-1fde-4a49-a34a-5dd40011b076_1 of rule 07b1ef73-1fde-4a49-a34a-5dd40011b076 and version 1 which had total 14 versions
Removed asset 07b5f85a-240f-11ed-b3d9-f661ea17fbce_104 of rule 07b5f85a-240f-11ed-b3d9-f661ea17fbce and version 104 which had total 6 versions
Removed asset 080bc66a-5d56-4d1f-8071-817671716db9_1 of rule 080bc66a-5d56-4d1f-8071-817671716db9 and version 1 which had total 10 versions
Removed asset 082e3f8c-6f80-485c-91eb-5b112cb79b28_100 of rule 082e3f8c-6f80-485c-91eb-5b112cb79b28 and version 100 which had total 9 versions
Removed asset 083fa162-e790-4d85-9aeb-4fea04188adb_1 of rule 083fa162-e790-4d85-9aeb-4fea04188adb and version 1 which had total 9 versions
Removed asset 0859355c-0f08-4b43-8ff5-7d2a4789fc08_1 of rule 0859355c-0f08-4b43-8ff5-7d2a4789fc08 and version 1 which had total 10 versions
Removed asset 089db1af-740d-4d84-9a5b-babd6de143b0_1 of rule 089db1af-740d-4d84-9a5b-babd6de143b0 and version 1 which had total 5 versions
Removed asset 092b068f-84ac-485d-8a55-7dd9e006715f_1 of rule 092b068f-84ac-485d-8a55-7dd9e006715f and version 1 which had total 11 versions
Removed asset 09443c92-46b3-45a4-8f25-383b028b258d_100 of rule 09443c92-46b3-45a4-8f25-383b028b258d and version 100 which had total 13 versions
Removed asset 095b6a58-8f88-4b59-827c-ab584ad4e759_1 of rule 095b6a58-8f88-4b59-827c-ab584ad4e759 and version 1 which had total 4 versions
Removed asset 09bc6c90-7501-494d-b015-5d988dc3f233_1 of rule 09bc6c90-7501-494d-b015-5d988dc3f233 and version 1 which had total 5 versions
Removed asset 09d028a5-dcde-409f-8ae0-557cef1b7082_1 of rule 09d028a5-dcde-409f-8ae0-557cef1b7082 and version 1 which had total 6 versions
Removed asset 0a97b20f-4144-49ea-be32-b540ecc445de_100 of rule 0a97b20f-4144-49ea-be32-b540ecc445de and version 100 which had total 6 versions
Removed asset 0ab319ef-92b8-4c7f-989b-5de93c852e93_1 of rule 0ab319ef-92b8-4c7f-989b-5de93c852e93 and version 1 which had total 5 versions
Removed asset 0abf0c5b-62dd-48d2-ac4e-6b43fe3a6e83_1 of rule 0abf0c5b-62dd-48d2-ac4e-6b43fe3a6e83 and version 1 which had total 11 versions
Removed asset 0b15bcad-aff1-4250-a5be-5d1b7eb56d07_1 of rule 0b15bcad-aff1-4250-a5be-5d1b7eb56d07 and version 1 which had total 4 versions
Removed asset 0b29cab4-dbbd-4a3f-9e8e-1287c7c11ae5_100 of rule 0b29cab4-dbbd-4a3f-9e8e-1287c7c11ae5 and version 100 which had total 11 versions
Removed asset 0b2f3da5-b5ec-47d1-908b-6ebb74814289_1 of rule 0b2f3da5-b5ec-47d1-908b-6ebb74814289 and version 1 which had total 18 versions
Removed asset 0b803267-74c5-444d-ae29-32b5db2d562a_1 of rule 0b803267-74c5-444d-ae29-32b5db2d562a and version 1 which had total 7 versions
Removed asset 0b96dfd8-5b8c-4485-9a1c-69ff7839786a_1 of rule 0b96dfd8-5b8c-4485-9a1c-69ff7839786a and version 1 which had total 6 versions
Removed asset 0c093569-dff9-42b6-87b1-0242d9f7d9b4_1 of rule 0c093569-dff9-42b6-87b1-0242d9f7d9b4 and version 1 which had total 2 versions
Removed asset 0c1e8fda-4f09-451e-bc77-a192b6cbfc32_1 of rule 0c1e8fda-4f09-451e-bc77-a192b6cbfc32 and version 1 which had total 2 versions
Removed asset 0c41e478-5263-4c69-8f9e-7dfd2c22da64_1 of rule 0c41e478-5263-4c69-8f9e-7dfd2c22da64 and version 1 which had total 7 versions
Removed asset 0c7ca5c2-728d-4ad9-b1c5-bbba83ecb1f4_100 of rule 0c7ca5c2-728d-4ad9-b1c5-bbba83ecb1f4 and version 100 which had total 19 versions
Removed asset 0cd2f3e6-41da-40e6-b28b-466f688f00a6_1 of rule 0cd2f3e6-41da-40e6-b28b-466f688f00a6 and version 1 which had total 4 versions
Removed asset 0ce6487d-8069-4888-9ddd-61b52490cebc_1 of rule 0ce6487d-8069-4888-9ddd-61b52490cebc and version 1 which had total 9 versions
Removed asset 0d160033-fab7-4e72-85a3-3a9d80c8bff7_1 of rule 0d160033-fab7-4e72-85a3-3a9d80c8bff7 and version 1 which had total 3 versions
Removed asset 0d69150b-96f8-467c-a86d-a67a3378ce77_10 of rule 0d69150b-96f8-467c-a86d-a67a3378ce77 and version 10 which had total 12 versions
Removed asset 0d8ad79f-9025-45d8-80c1-4f0cd3c5e8e5_100 of rule 0d8ad79f-9025-45d8-80c1-4f0cd3c5e8e5 and version 100 which had total 15 versions
Removed asset 0e4367a0-a483-439d-ad2e-d90500b925fd_1 of rule 0e4367a0-a483-439d-ad2e-d90500b925fd and version 1 which had total 4 versions
Removed asset 0e52157a-8e96-4a95-a6e3-5faae5081a74_100 of rule 0e52157a-8e96-4a95-a6e3-5faae5081a74 and version 100 which had total 8 versions

@Mikaayenson This ensure no rules with one version is removed.

Also this opens up a possibility of another angle, where we could pick

  • Older versions, with count exceeding say 6 versions can be trimmed, going by the below analysis
  • But there is a high risk of not so old rules, having regular tunings can fall under the axe
Image

The trimming would some like this.

❯ python -m detection_rules dev build-release 
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building package 8.18
 - 4 rules excluded from package
Rule changes detected!
 - 951 changed rules
 - 1 new rules
 - 0 newly deprecated rules
run `build-release --update-version-lock` to update version.lock.json and deprecated_rules.json
Package saved to: /Users/shashankks/elastic_workspace/detection-rules/releases/8.18
loaded security_detection_engine manifests from the following package versions: ['8.17.3', '8.17.2', '8.17.1', '8.16.5', '8.16.4', '8.16.3', '8.16.2', '8.16.1', '8.15.14', '8.15.13', '8.15.12', '8.15.11', '8.15.10', '8.15.9', '8.15.8', '8.15.7', '8.15.6', '8.15.5', '8.15.4', '8.15.3', '8.15.2', '8.15.1', '8.14.20', '8.14.19', '8.14.18', '8.14.17', '8.14.16', '8.14.15', '8.14.14', '8.14.13', '8.14.12', '8.14.11', '8.14.10', '8.14.9', '8.14.8', '8.14.7', '8.14.6', '8.14.5', '8.14.4', '8.14.3', '8.14.2', '8.14.1', '8.13.23', '8.13.22', '8.13.21', '8.13.20', '8.13.19', '8.13.18', '8.13.17', '8.13.16', '8.13.15', '8.13.14', '8.13.13', '8.13.12', '8.13.11', '8.13.10', '8.13.9', '8.13.8', '8.13.7', '8.13.6', '8.13.5', '8.13.4', '8.13.3', '8.13.2', '8.13.1', '8.12.26', '8.12.25', '8.12.24', '8.12.23', '8.12.22', '8.12.21', '8.12.20', '8.12.19', '8.12.18', '8.12.17', '8.12.16', '8.12.15', '8.12.14', '8.12.13', '8.12.12', '8.12.11', '8.12.10', '8.12.9', '8.12.8', '8.12.7', '8.12.6', '8.12.5', '8.12.4', '8.12.3', '8.12.2', '8.12.1', '8.11.21', '8.11.20', '8.11.19', '8.11.18', '8.11.17', '8.11.16', '8.11.15', '8.11.14', '8.11.13', '8.11.12', '8.11.11', '8.11.10', '8.11.9', '8.11.8', '8.11.7', '8.11.6', '8.11.5', '8.11.4', '8.11.3', '8.11.2', '8.11.1', '8.10.18', '8.10.17', '8.10.16', '8.10.15', '8.10.14', '8.10.13', '8.10.12', '8.10.11', '8.10.10', '8.10.9', '8.10.8', '8.10.7', '8.10.6', '8.10.5', '8.10.4', '8.10.3', '8.10.2', '8.10.1', '8.9.15', '8.9.14', '8.9.13', '8.9.12', '8.9.11', '8.9.10', '8.9.9', '8.9.8', '8.9.7', '8.9.6', '8.9.5', '8.9.4', '8.9.3', '8.9.2', '8.9.1', '8.8.15', '8.8.14', '8.8.13', '8.8.12', '8.8.11', '8.8.10', '8.8.9', '8.8.8', '8.8.7', '8.8.6', '8.8.5', '8.8.4', '8.8.3', '8.8.2', '8.8.1', '8.7.13', '8.7.12', '8.7.11', '8.7.10', '8.7.9', '8.7.8', '8.7.7', '8.7.6', '8.7.5', '8.7.4', '8.7.3', '8.7.2', '8.7.1', '8.6.10', '8.6.9', '8.6.8', '8.6.7', '8.6.6', '8.6.5', '8.6.4', '8.6.3', '8.6.2', '8.6.1', '8.5.8', '8.5.7', '8.5.6', '8.5.5', '8.5.4', '8.5.3', '8.5.2', '8.5.1', '8.4.5', '8.4.4', '8.4.3', '8.4.2', '8.4.1', '8.3.4', '8.3.3', '8.3.2', '8.3.1', '8.2.1', '8.1.1', '1.0.2', '1.0.1']
[+] Adding all historical rule versions in our release package for version             8.18.0
Total number of rules: 11578
Excess count: 78
Removed asset 000047bb-b27a-47ec-8b62-ef1a5d2c9e19_100 of rule 000047bb-b27a-47ec-8b62-ef1a5d2c9e19 and version 100 which had total 17 versions
Removed asset 00140285-b827-4aee-aa09-8113f58a08f3_100 of rule 00140285-b827-4aee-aa09-8113f58a08f3 and version 100 which had total 23 versions
Removed asset 0022d47d-39c7-4f69-a232-4fe9dc7a3acd_10 of rule 0022d47d-39c7-4f69-a232-4fe9dc7a3acd and version 10 which had total 22 versions
Removed asset 0136b315-b566-482f-866c-1d8e2477ba16_1 of rule 0136b315-b566-482f-866c-1d8e2477ba16 and version 1 which had total 9 versions
Removed asset 015cca13-8832-49ac-a01b-a396114809f6_1 of rule 015cca13-8832-49ac-a01b-a396114809f6 and version 1 which had total 10 versions
Removed asset 0171f283-ade7-4f87-9521-ac346c68cc9b_1 of rule 0171f283-ade7-4f87-9521-ac346c68cc9b and version 1 which had total 7 versions
Removed asset 027ff9ea-85e7-42e3-99d2-bbb7069e02eb_1 of rule 027ff9ea-85e7-42e3-99d2-bbb7069e02eb and version 1 which had total 12 versions
Removed asset 02a4576a-7480-4284-9327-548a806b5e48_100 of rule 02a4576a-7480-4284-9327-548a806b5e48 and version 100 which had total 14 versions
Removed asset 02ea4563-ec10-4974-b7de-12e65aa4f9b3_1 of rule 02ea4563-ec10-4974-b7de-12e65aa4f9b3 and version 1 which had total 9 versions
Removed asset 03024bd9-d23f-4ec1-8674-3cf1a21e130b_100 of rule 03024bd9-d23f-4ec1-8674-3cf1a21e130b and version 100 which had total 9 versions
Removed asset 035889c4-2686-4583-a7df-67f89c292f2c_100 of rule 035889c4-2686-4583-a7df-67f89c292f2c and version 100 which had total 18 versions
Removed asset 0415f22a-2336-45fa-ba07-618a5942e22c_1 of rule 0415f22a-2336-45fa-ba07-618a5942e22c and version 1 which had total 13 versions
Removed asset 053a0387-f3b5-4ba5-8245-8002cca2bd08_100 of rule 053a0387-f3b5-4ba5-8245-8002cca2bd08 and version 100 which had total 16 versions
Removed asset 054db96b-fd34-43b3-9af2-587b3bd33964_1 of rule 054db96b-fd34-43b3-9af2-587b3bd33964 and version 1 which had total 7 versions
Removed asset 0564fb9d-90b9-4234-a411-82a546dc1343_100 of rule 0564fb9d-90b9-4234-a411-82a546dc1343 and version 100 which had total 19 versions
Removed asset 05b358de-aa6d-4f6c-89e6-78f74018b43b_100 of rule 05b358de-aa6d-4f6c-89e6-78f74018b43b and version 100 which had total 17 versions
Removed asset 05e5a668-7b51-4a67-93ab-e9af405c9ef3_100 of rule 05e5a668-7b51-4a67-93ab-e9af405c9ef3 and version 100 which had total 11 versions
Removed asset 0635c542-1b96-4335-9b47-126582d2c19a_100 of rule 0635c542-1b96-4335-9b47-126582d2c19a and version 100 which had total 20 versions
Removed asset 06568a02-af29-4f20-929c-f3af281e41aa_1 of rule 06568a02-af29-4f20-929c-f3af281e41aa and version 1 which had total 11 versions
Removed asset 06a7a03c-c735-47a6-a313-51c354aef6c3_1 of rule 06a7a03c-c735-47a6-a313-51c354aef6c3 and version 1 which had total 12 versions
Removed asset 06dceabf-adca-48af-ac79-ffdf4c3b1e9a_100 of rule 06dceabf-adca-48af-ac79-ffdf4c3b1e9a and version 100 which had total 19 versions
Removed asset 074464f9-f30d-4029-8c03-0ed237fffec7_100 of rule 074464f9-f30d-4029-8c03-0ed237fffec7 and version 100 which had total 19 versions
Removed asset 0787daa6-f8c5-453b-a4ec-048037f6c1cd_1 of rule 0787daa6-f8c5-453b-a4ec-048037f6c1cd and version 1 which had total 7 versions
Removed asset 07b1ef73-1fde-4a49-a34a-5dd40011b076_1 of rule 07b1ef73-1fde-4a49-a34a-5dd40011b076 and version 1 which had total 14 versions
Removed asset 080bc66a-5d56-4d1f-8071-817671716db9_1 of rule 080bc66a-5d56-4d1f-8071-817671716db9 and version 1 which had total 10 versions
Removed asset 082e3f8c-6f80-485c-91eb-5b112cb79b28_100 of rule 082e3f8c-6f80-485c-91eb-5b112cb79b28 and version 100 which had total 9 versions
Removed asset 083fa162-e790-4d85-9aeb-4fea04188adb_1 of rule 083fa162-e790-4d85-9aeb-4fea04188adb and version 1 which had total 9 versions
Removed asset 0859355c-0f08-4b43-8ff5-7d2a4789fc08_1 of rule 0859355c-0f08-4b43-8ff5-7d2a4789fc08 and version 1 which had total 10 versions
Removed asset 092b068f-84ac-485d-8a55-7dd9e006715f_1 of rule 092b068f-84ac-485d-8a55-7dd9e006715f and version 1 which had total 11 versions
Removed asset 09443c92-46b3-45a4-8f25-383b028b258d_100 of rule 09443c92-46b3-45a4-8f25-383b028b258d and version 100 which had total 13 versions
Removed asset 0abf0c5b-62dd-48d2-ac4e-6b43fe3a6e83_1 of rule 0abf0c5b-62dd-48d2-ac4e-6b43fe3a6e83 and version 1 which had total 11 versions
Removed asset 0b29cab4-dbbd-4a3f-9e8e-1287c7c11ae5_100 of rule 0b29cab4-dbbd-4a3f-9e8e-1287c7c11ae5 and version 100 which had total 11 versions
Removed asset 0b2f3da5-b5ec-47d1-908b-6ebb74814289_1 of rule 0b2f3da5-b5ec-47d1-908b-6ebb74814289 and version 1 which had total 18 versions
Removed asset 0b803267-74c5-444d-ae29-32b5db2d562a_1 of rule 0b803267-74c5-444d-ae29-32b5db2d562a and version 1 which had total 7 versions
Removed asset 0c41e478-5263-4c69-8f9e-7dfd2c22da64_1 of rule 0c41e478-5263-4c69-8f9e-7dfd2c22da64 and version 1 which had total 7 versions
Removed asset 0c7ca5c2-728d-4ad9-b1c5-bbba83ecb1f4_100 of rule 0c7ca5c2-728d-4ad9-b1c5-bbba83ecb1f4 and version 100 which had total 19 versions
Removed asset 0ce6487d-8069-4888-9ddd-61b52490cebc_1 of rule 0ce6487d-8069-4888-9ddd-61b52490cebc and version 1 which had total 9 versions
Removed asset 0d69150b-96f8-467c-a86d-a67a3378ce77_10 of rule 0d69150b-96f8-467c-a86d-a67a3378ce77 and version 10 which had total 12 versions
Removed asset 0d8ad79f-9025-45d8-80c1-4f0cd3c5e8e5_100 of rule 0d8ad79f-9025-45d8-80c1-4f0cd3c5e8e5 and version 100 which had total 15 versions
Removed asset 0e52157a-8e96-4a95-a6e3-5faae5081a74_100 of rule 0e52157a-8e96-4a95-a6e3-5faae5081a74 and version 100 which had total 8 versions
(.venv) 
detection-rules on  main [$!?] is 📦 v0.4.3 via 🐍 v3.12.8 (.venv) on ☁️  [email protected] took 1m53s 
❯

cc @eric-forte-elastic / @traut for thoughts

@shashank-elastic
Copy link
Contributor Author

Spot check the version Count from the current 8.17.3 assets

For Rule 000047bb-b27a-47ec-8b62-ef1a5d2c9e19 17 versions are reported and 17 found

Image

For rule 0c7ca5c2-728d-4ad9-b1c5-bbba83ecb1f4 19 versions are reported and 19 found
Image

@shashank-elastic
Copy link
Contributor Author

PR in Draft Mode with these observations

  • For testing purpose we continue to use 11500 as our asset limit
python -m detection_rules dev build-release 
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building package 8.18
 - 4 rules excluded from package
Rule changes detected!
 - 962 changed rules
 - 1 new rules
 - 0 newly deprecated rules
run `build-release --update-version-lock` to update version.lock.json and deprecated_rules.json
Package saved to: /Users/shashankks/elastic_workspace/detection-rules/releases/8.18
loaded security_detection_engine manifests from the following package versions: ['8.17.3', '8.17.2', '8.17.1', '8.16.5', '8.16.4', '8.16.3', '8.16.2', '8.16.1', '8.15.14', '8.15.13', '8.15.12', '8.15.11', '8.15.10', '8.15.9', '8.15.8', '8.15.7', '8.15.6', '8.15.5', '8.15.4', '8.15.3', '8.15.2', '8.15.1', '8.14.20', '8.14.19', '8.14.18', '8.14.17', '8.14.16', '8.14.15', '8.14.14', '8.14.13', '8.14.12', '8.14.11', '8.14.10', '8.14.9', '8.14.8', '8.14.7', '8.14.6', '8.14.5', '8.14.4', '8.14.3', '8.14.2', '8.14.1', '8.13.23', '8.13.22', '8.13.21', '8.13.20', '8.13.19', '8.13.18', '8.13.17', '8.13.16', '8.13.15', '8.13.14', '8.13.13', '8.13.12', '8.13.11', '8.13.10', '8.13.9', '8.13.8', '8.13.7', '8.13.6', '8.13.5', '8.13.4', '8.13.3', '8.13.2', '8.13.1', '8.12.26', '8.12.25', '8.12.24', '8.12.23', '8.12.22', '8.12.21', '8.12.20', '8.12.19', '8.12.18', '8.12.17', '8.12.16', '8.12.15', '8.12.14', '8.12.13', '8.12.12', '8.12.11', '8.12.10', '8.12.9', '8.12.8', '8.12.7', '8.12.6', '8.12.5', '8.12.4', '8.12.3', '8.12.2', '8.12.1', '8.11.21', '8.11.20', '8.11.19', '8.11.18', '8.11.17', '8.11.16', '8.11.15', '8.11.14', '8.11.13', '8.11.12', '8.11.11', '8.11.10', '8.11.9', '8.11.8', '8.11.7', '8.11.6', '8.11.5', '8.11.4', '8.11.3', '8.11.2', '8.11.1', '8.10.18', '8.10.17', '8.10.16', '8.10.15', '8.10.14', '8.10.13', '8.10.12', '8.10.11', '8.10.10', '8.10.9', '8.10.8', '8.10.7', '8.10.6', '8.10.5', '8.10.4', '8.10.3', '8.10.2', '8.10.1', '8.9.15', '8.9.14', '8.9.13', '8.9.12', '8.9.11', '8.9.10', '8.9.9', '8.9.8', '8.9.7', '8.9.6', '8.9.5', '8.9.4', '8.9.3', '8.9.2', '8.9.1', '8.8.15', '8.8.14', '8.8.13', '8.8.12', '8.8.11', '8.8.10', '8.8.9', '8.8.8', '8.8.7', '8.8.6', '8.8.5', '8.8.4', '8.8.3', '8.8.2', '8.8.1', '8.7.13', '8.7.12', '8.7.11', '8.7.10', '8.7.9', '8.7.8', '8.7.7', '8.7.6', '8.7.5', '8.7.4', '8.7.3', '8.7.2', '8.7.1', '8.6.10', '8.6.9', '8.6.8', '8.6.7', '8.6.6', '8.6.5', '8.6.4', '8.6.3', '8.6.2', '8.6.1', '8.5.8', '8.5.7', '8.5.6', '8.5.5', '8.5.4', '8.5.3', '8.5.2', '8.5.1', '8.4.5', '8.4.4', '8.4.3', '8.4.2', '8.4.1', '8.3.4', '8.3.3', '8.3.2', '8.3.1', '8.2.1', '8.1.1', '1.0.2', '1.0.1']
[+] Adding all historical rule versions in our release package for version             8.18.0
[!] Asset count exceeded, applying smart limits to historical rule versions
Removed asset 000047bb-b27a-47ec-8b62-ef1a5d2c9e19_100 of rule 000047bb-b27a-47ec-8b62-ef1a5d2c9e19 and version 100 which had total 17 versions
Removed asset 00140285-b827-4aee-aa09-8113f58a08f3_100 of rule 00140285-b827-4aee-aa09-8113f58a08f3 and version 100 which had total 23 versions
Removed asset 0022d47d-39c7-4f69-a232-4fe9dc7a3acd_10 of rule 0022d47d-39c7-4f69-a232-4fe9dc7a3acd and version 10 which had total 22 versions
Removed asset 00678712-b2df-11ed-afe9-f661ea17fbcc_1 of rule 00678712-b2df-11ed-afe9-f661ea17fbcc and version 1 which had total 3 versions
Removed asset 0136b315-b566-482f-866c-1d8e2477ba16_1 of rule 0136b315-b566-482f-866c-1d8e2477ba16 and version 1 which had total 9 versions
Removed asset 015cca13-8832-49ac-a01b-a396114809f6_1 of rule 015cca13-8832-49ac-a01b-a396114809f6 and version 1 which had total 10 versions
Removed asset 0171f283-ade7-4f87-9521-ac346c68cc9b_1 of rule 0171f283-ade7-4f87-9521-ac346c68cc9b and version 1 which had total 7 versions
Removed asset 01c49712-25bc-49d2-a27d-d7ce52f5dc49_1 of rule 01c49712-25bc-49d2-a27d-d7ce52f5dc49 and version 1 which had total 4 versions
Removed asset 027ff9ea-85e7-42e3-99d2-bbb7069e02eb_1 of rule 027ff9ea-85e7-42e3-99d2-bbb7069e02eb and version 1 which had total 12 versions
Removed asset 0294f105-d7af-4a02-ae90-35f56763ffa2_1 of rule 0294f105-d7af-4a02-ae90-35f56763ffa2 and version 1 which had total 4 versions
Removed asset 02a23ee7-c8f8-4701-b99d-e9038ce313cb_1 of rule 02a23ee7-c8f8-4701-b99d-e9038ce313cb and version 1 which had total 6 versions
Removed asset 02a4576a-7480-4284-9327-548a806b5e48_100 of rule 02a4576a-7480-4284-9327-548a806b5e48 and version 100 which had total 14 versions
Removed asset 02bab13d-fb14-4d7c-b6fe-4a28874d37c5_1 of rule 02bab13d-fb14-4d7c-b6fe-4a28874d37c5 and version 1 which had total 3 versions
Removed asset 02ea4563-ec10-4974-b7de-12e65aa4f9b3_1 of rule 02ea4563-ec10-4974-b7de-12e65aa4f9b3 and version 1 which had total 9 versions
Removed asset 03024bd9-d23f-4ec1-8674-3cf1a21e130b_100 of rule 03024bd9-d23f-4ec1-8674-3cf1a21e130b and version 100 which had total 9 versions
Removed asset 035889c4-2686-4583-a7df-67f89c292f2c_100 of rule 035889c4-2686-4583-a7df-67f89c292f2c and version 100 which had total 18 versions
Removed asset 035a6f21-4092-471d-9cda-9e379f459b1e_1 of rule 035a6f21-4092-471d-9cda-9e379f459b1e and version 1 which had total 3 versions
Removed asset 0369e8a6-0fa7-4e7a-961a-53180a4c966e_1 of rule 0369e8a6-0fa7-4e7a-961a-53180a4c966e and version 1 which had total 3 versions
Removed asset 03a514d9-500e-443e-b6a9-72718c548f6c_1 of rule 03a514d9-500e-443e-b6a9-72718c548f6c and version 1 which had total 2 versions
Removed asset 03c23d45-d3cb-4ad4-ab5d-b361ffe8724a_1 of rule 03c23d45-d3cb-4ad4-ab5d-b361ffe8724a and version 1 which had total 3 versions
Removed asset 0415f22a-2336-45fa-ba07-618a5942e22c_1 of rule 0415f22a-2336-45fa-ba07-618a5942e22c and version 1 which had total 13 versions
Removed asset 043d80a3-c49e-43ef-9c72-1088f0c7b278_1 of rule 043d80a3-c49e-43ef-9c72-1088f0c7b278 and version 1 which had total 6 versions
Removed asset 04c5a96f-19c5-44fd-9571-a0b033f9086f_100 of rule 04c5a96f-19c5-44fd-9571-a0b033f9086f and version 100 which had total 5 versions
Removed asset 053a0387-f3b5-4ba5-8245-8002cca2bd08_100 of rule 053a0387-f3b5-4ba5-8245-8002cca2bd08 and version 100 which had total 16 versions
Removed asset 054db96b-fd34-43b3-9af2-587b3bd33964_1 of rule 054db96b-fd34-43b3-9af2-587b3bd33964 and version 1 which had total 7 versions
Removed asset 0564fb9d-90b9-4234-a411-82a546dc1343_100 of rule 0564fb9d-90b9-4234-a411-82a546dc1343 and version 100 which had total 19 versions
Removed asset 05b358de-aa6d-4f6c-89e6-78f74018b43b_100 of rule 05b358de-aa6d-4f6c-89e6-78f74018b43b and version 100 which had total 17 versions
Removed asset 05cad2fb-200c-407f-b472-02ea8c9e5e4a_1 of rule 05cad2fb-200c-407f-b472-02ea8c9e5e4a and version 1 which had total 4 versions
Removed asset 05e5a668-7b51-4a67-93ab-e9af405c9ef3_100 of rule 05e5a668-7b51-4a67-93ab-e9af405c9ef3 and version 100 which had total 11 versions
Removed asset 0635c542-1b96-4335-9b47-126582d2c19a_100 of rule 0635c542-1b96-4335-9b47-126582d2c19a and version 100 which had total 20 versions
Removed asset 06568a02-af29-4f20-929c-f3af281e41aa_1 of rule 06568a02-af29-4f20-929c-f3af281e41aa and version 1 which had total 11 versions
Removed asset 0678bc9c-b71a-433b-87e6-2f664b6b3131_1 of rule 0678bc9c-b71a-433b-87e6-2f664b6b3131 and version 1 which had total 4 versions
Removed asset 06a7a03c-c735-47a6-a313-51c354aef6c3_1 of rule 06a7a03c-c735-47a6-a313-51c354aef6c3 and version 1 which had total 12 versions
Removed asset 06d555e4-c8ce-4d90-90e1-ec7f66df5a6a_1 of rule 06d555e4-c8ce-4d90-90e1-ec7f66df5a6a and version 1 which had total 2 versions
Removed asset 06dceabf-adca-48af-ac79-ffdf4c3b1e9a_100 of rule 06dceabf-adca-48af-ac79-ffdf4c3b1e9a and version 100 which had total 19 versions
Removed asset 074464f9-f30d-4029-8c03-0ed237fffec7_100 of rule 074464f9-f30d-4029-8c03-0ed237fffec7 and version 100 which had total 19 versions
Removed asset 07639887-da3a-4fbf-9532-8ce748ff8c50_1 of rule 07639887-da3a-4fbf-9532-8ce748ff8c50 and version 1 which had total 6 versions
Removed asset 0787daa6-f8c5-453b-a4ec-048037f6c1cd_1 of rule 0787daa6-f8c5-453b-a4ec-048037f6c1cd and version 1 which had total 7 versions
Removed asset 07b1ef73-1fde-4a49-a34a-5dd40011b076_1 of rule 07b1ef73-1fde-4a49-a34a-5dd40011b076 and version 1 which had total 14 versions
Removed asset 07b5f85a-240f-11ed-b3d9-f661ea17fbce_104 of rule 07b5f85a-240f-11ed-b3d9-f661ea17fbce and version 104 which had total 6 versions
Removed asset 080bc66a-5d56-4d1f-8071-817671716db9_1 of rule 080bc66a-5d56-4d1f-8071-817671716db9 and version 1 which had total 10 versions
Removed asset 082e3f8c-6f80-485c-91eb-5b112cb79b28_100 of rule 082e3f8c-6f80-485c-91eb-5b112cb79b28 and version 100 which had total 9 versions
Removed asset 083fa162-e790-4d85-9aeb-4fea04188adb_1 of rule 083fa162-e790-4d85-9aeb-4fea04188adb and version 1 which had total 9 versions
Removed asset 0859355c-0f08-4b43-8ff5-7d2a4789fc08_1 of rule 0859355c-0f08-4b43-8ff5-7d2a4789fc08 and version 1 which had total 10 versions
Removed asset 089db1af-740d-4d84-9a5b-babd6de143b0_1 of rule 089db1af-740d-4d84-9a5b-babd6de143b0 and version 1 which had total 5 versions
Removed asset 092b068f-84ac-485d-8a55-7dd9e006715f_1 of rule 092b068f-84ac-485d-8a55-7dd9e006715f and version 1 which had total 11 versions
Removed asset 09443c92-46b3-45a4-8f25-383b028b258d_100 of rule 09443c92-46b3-45a4-8f25-383b028b258d and version 100 which had total 13 versions
Removed asset 095b6a58-8f88-4b59-827c-ab584ad4e759_1 of rule 095b6a58-8f88-4b59-827c-ab584ad4e759 and version 1 which had total 4 versions
Removed asset 09bc6c90-7501-494d-b015-5d988dc3f233_1 of rule 09bc6c90-7501-494d-b015-5d988dc3f233 and version 1 which had total 5 versions
Removed asset 09d028a5-dcde-409f-8ae0-557cef1b7082_1 of rule 09d028a5-dcde-409f-8ae0-557cef1b7082 and version 1 which had total 6 versions
Removed asset 0a97b20f-4144-49ea-be32-b540ecc445de_100 of rule 0a97b20f-4144-49ea-be32-b540ecc445de and version 100 which had total 6 versions
Removed asset 0ab319ef-92b8-4c7f-989b-5de93c852e93_1 of rule 0ab319ef-92b8-4c7f-989b-5de93c852e93 and version 1 which had total 5 versions
Removed asset 0abf0c5b-62dd-48d2-ac4e-6b43fe3a6e83_1 of rule 0abf0c5b-62dd-48d2-ac4e-6b43fe3a6e83 and version 1 which had total 11 versions
Removed asset 0b15bcad-aff1-4250-a5be-5d1b7eb56d07_1 of rule 0b15bcad-aff1-4250-a5be-5d1b7eb56d07 and version 1 which had total 4 versions
Removed asset 0b29cab4-dbbd-4a3f-9e8e-1287c7c11ae5_100 of rule 0b29cab4-dbbd-4a3f-9e8e-1287c7c11ae5 and version 100 which had total 11 versions
Removed asset 0b2f3da5-b5ec-47d1-908b-6ebb74814289_1 of rule 0b2f3da5-b5ec-47d1-908b-6ebb74814289 and version 1 which had total 18 versions
Removed asset 0b803267-74c5-444d-ae29-32b5db2d562a_1 of rule 0b803267-74c5-444d-ae29-32b5db2d562a and version 1 which had total 7 versions
Removed asset 0b96dfd8-5b8c-4485-9a1c-69ff7839786a_1 of rule 0b96dfd8-5b8c-4485-9a1c-69ff7839786a and version 1 which had total 6 versions
Removed asset 0c093569-dff9-42b6-87b1-0242d9f7d9b4_1 of rule 0c093569-dff9-42b6-87b1-0242d9f7d9b4 and version 1 which had total 2 versions
Removed asset 0c1e8fda-4f09-451e-bc77-a192b6cbfc32_1 of rule 0c1e8fda-4f09-451e-bc77-a192b6cbfc32 and version 1 which had total 2 versions
Removed asset 0c41e478-5263-4c69-8f9e-7dfd2c22da64_1 of rule 0c41e478-5263-4c69-8f9e-7dfd2c22da64 and version 1 which had total 7 versions
Removed asset 0c7ca5c2-728d-4ad9-b1c5-bbba83ecb1f4_100 of rule 0c7ca5c2-728d-4ad9-b1c5-bbba83ecb1f4 and version 100 which had total 19 versions
Removed asset 0cd2f3e6-41da-40e6-b28b-466f688f00a6_1 of rule 0cd2f3e6-41da-40e6-b28b-466f688f00a6 and version 1 which had total 4 versions
Removed asset 0ce6487d-8069-4888-9ddd-61b52490cebc_1 of rule 0ce6487d-8069-4888-9ddd-61b52490cebc and version 1 which had total 9 versions
Removed asset 0d160033-fab7-4e72-85a3-3a9d80c8bff7_1 of rule 0d160033-fab7-4e72-85a3-3a9d80c8bff7 and version 1 which had total 3 versions
Removed asset 0d69150b-96f8-467c-a86d-a67a3378ce77_10 of rule 0d69150b-96f8-467c-a86d-a67a3378ce77 and version 10 which had total 12 versions
Removed asset 0d8ad79f-9025-45d8-80c1-4f0cd3c5e8e5_100 of rule 0d8ad79f-9025-45d8-80c1-4f0cd3c5e8e5 and version 100 which had total 15 versions
Removed asset 0e4367a0-a483-439d-ad2e-d90500b925fd_1 of rule 0e4367a0-a483-439d-ad2e-d90500b925fd and version 1 which had total 4 versions
Removed asset 0e52157a-8e96-4a95-a6e3-5faae5081a74_100 of rule 0e52157a-8e96-4a95-a6e3-5faae5081a74 and version 100 which had total 8 versions
[+] Adding historical rules from 8.17.3 package
- sha256: 83c5e6c046ab46aa47bcf8e50fb56699138277a26379ebee920e01a207a3a1c7
- 1310 rules included
(.venv)
  • Once the Package is built lets walk through the limit of rule 0e52157a-8e96-4a95-a6e3-5faae5081a74
  • Rule 0e52157a-8e96-4a95-a6e3-5faae5081a74 originally had 8 versions and was one of the older rules whose version 0e52157a-8e96-4a95-a6e3-5faae5081a74_100 asset was deleted
  • The same is reflected in the local release files , you cannot find 0e52157a-8e96-4a95-a6e3-5faae5081a74_100
Image
  • There is a new version added 0e52157a-8e96-4a95-a6e3-5faae5081a74_207 as opposed to latest version in 8.17.3
Image
  • This is a new version on main with investigation guides see diff
Image
  • Now Lets look at the journey of the rule
    • Rule was originally written via PR Jan 2022
    • Originally min stacked to 8.3 PR
    • The min stacked moved to 8.8.0 to incorporate integration changes via PR April 2024
    • Bacport versions trimmed when support was removed for 8.8 via PR May 2024
    • Investigation Guides added.
  • With the rule journey tracking removing the 0e52157a-8e96-4a95-a6e3-5faae5081a74_100 is very very safe as we no longer support updates to those stacks.

Why the PR is in draft mode - I really like the verbose data that I have used to call out version removal. I would want it to be part of the smart_limits.txt which we can make it part of releases/8.18/extras where we will have a documented version of it when the trimming happens. I dont want a console output becuase if the trimming is huge, the output is extremly large which is like err...

@shashank-elastic shashank-elastic linked a pull request Jan 24, 2025 that will close this issue
Draft
5 tasks
@shashank-elastic shashank-elastic linked a pull request Jan 24, 2025 that will close this issue
Enable Smart Limits for Rule Versions #4419
Draft
5 tasks
@Mikaayenson
Copy link
Contributor

You probably should take a closer look at the logic. For rule 000047bb-b27a-47ec-8b62-ef1a5d2c9e19 it shouldnt be removing version 100. It looks like the oldest would be 6 assuming thats in your local data like it is in the integrations repo.

Image

@shashank-elastic
Copy link
Contributor Author

You probably should take a closer look at the logic. For rule 000047bb-b27a-47ec-8b62-ef1a5d2c9e19 it shouldnt be removing version 100. It looks like the oldest would be 6 assuming thats in your local data like it is in the integrations repo.

Image

@Mikaayenson Not necessarily right.

Assuming we have forked at version 1 so we get the version 100 and then there are updates for rule fork and older supported version. 6 would be latest than 100 :) Well i did crack my head in the afternoon for the same. if we had forked at version 6 then fork version would be 106 making it latest than base version 6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shashank-elastic shashank-elastic

Labels
community enhancement New feature or request Team: TRADE
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enable Smart Limits for Rule Versions elastic/detection-rules
2 participants
@Mikaayenson @shashank-elastic