Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollover + alias E2E testing #53

Closed
16 tasks done
Tracked by #18999
AlexRuiz7 opened this issue Nov 13, 2023 · 2 comments
Closed
16 tasks done
Tracked by #18999

Rollover + alias E2E testing #53

AlexRuiz7 opened this issue Nov 13, 2023 · 2 comments
Assignees
Labels
level/task Task issue request/operational Operational requests type/test Test issue

Comments

@AlexRuiz7
Copy link
Member

AlexRuiz7 commented Nov 13, 2023

Test description

In order to test wazuh/wazuh#18999, these manual tests must pass:

  • 01 - AIO installation (assistant)
  • 02 - AIO installation (step-by-step)
  • 03 - Cluster installation (assistant)
  • 04 - Cluster installation (step-by-step)
  • 05 - AIO upgrade without existing rollover policy
  • 06 - Cluster upgrade without existing rollover policy
  • 07 - AIO upgrade with existing rollover policy
  • 08 - Cluster upgrade with existing rollover policy
  • 09 - Installation alternatives: OVA
  • 10 - Installation alternatives: AMI
  • 11 - Installation alternatives: Docker
  • 12 - Installation alternatives: Kubernetes
  • 13 - Installation alternatives: Offline
  • 14 - Installation alternatives: Ansible
  • 15 - Installation alternatives: Puppet

*AIO: all-in-one

All methods must verify that:

  • Using the documentation for each deployed method, the installation is successful.

    Details

    The installation completes without errors.

  • The ISM policy rollover_policy is created*.

    Details

    Use GET _plugins/_ism/policies/rollover_policy to verify the policy exists.

  • The wazuh-alerts and wazuh-archives index templates are created.

    Details

    Use GET _template/wazuh*rollover to verify the templates exist.

  • The wazuh-alerts and wazuh-archives aliases are created.

    Details

    Use GET _alias/wazuh-a* to verify the aliases exist.

  • The initial indices are created and being managed by the rollover_policy policy*.

    Details

    For tests number 07 and 08, see the special checks below.

    Use GET _plugins/_ism/explain/wazuh-*-000001 to check if the indices are being managed by the policy.

  • The manage_ism action group and role are created.

    Details

    Use GET _plugins/_security/api/roles/manage_ism to check if the action group and role exists.

    {
        "manage_ism": {
            "reserved": true,
            "hidden": false,
            "cluster_permissions": [
            "manage_ism"
            ],
            "index_permissions": [],
            "tenant_permissions": [],
            "static": false
        }
    }

For tests number 07 and 08, these additional checks must be performed:

  • The policy with the highest priority takes precedence to manage the indices.

    Details

    Steps:

    • Install wazuh-indexer <4.8.0
    • Create a rollover policy to manage the wazuh-alerts and wazuh-archives indices.
    • Upgrade to wazuh-indexer 4.8.0
    • Initialize the cluster with a custom priority for the policy, depending on the scenario.
      • Scenario A:
        • Procedure: create the policy with a LOWER priority than the existing rollover policy.
        • Expected result: the policy IS created and DOES NOT take precedence in new indices.
      • Scenario B:
        • Procedure: create the policy with a EQUAL priority than the existing rollover policy.
        • Expected result: the policy IS NOT created (expected error).
      • Scenario C:
        • Procedure: create the policy with a HIGHER priority than the existing rollover policy.
        • Expected result: the policy IS created and DOES take precedence in new indices.

Test report procedure

All test results must have one of the following statuses:

🟢 All checks passed.
🔴 There is at least one failed result.
🟡 There is at least one expected failure or skipped test and no failures.
Not started.
Skipped.

Any failing test must be properly addressed with a new issue, detailing the error and the possible cause.

An extended report of the test results must be attached as a ZIP or TXT file. Please attach any documents, screenshots, or tables to the issue update with the results. This report can be used by the auditors to dig deeper into any possible failures and details.

Conclusions

All tests have been executed and the results can be found here.

Status Test Failure type Notes
01 Covered*
02 Covered*
🟢 03
🟢 04
05 Covered*
🟢 06
07 Covered*
🔴 08
09 Covered*
10 Covered*
11 Covered*
12 Covered*
13 Covered*
14 Covered*
15 Covered*
  • Single-node (AIO) tests are covered by the multi-node tests.
  • Installation alternatives are based on DEB / RPM packages. Covered in multi-node tests.

All tests have passed and the fails have been reported or justified. Therefore, I conclude that this issue is finished and OK for this release candidate.

Auditors validation

The definition of done for this one is the validation of the conclusions and the test results from all auditors.

All checks from below must be accepted in order to close this issue.

  • @wazuh/indexer
@AlexRuiz7 AlexRuiz7 added level/task Task issue type/test Test issue labels Nov 13, 2023
@AlexRuiz7 AlexRuiz7 self-assigned this Nov 13, 2023
@AlexRuiz7
Copy link
Member Author

AlexRuiz7 commented Nov 13, 2023

Test results

config.yml

nodes:
    # Wazuh indexer nodes
    indexer:
        - name: node-1
          ip: "192.168.56.10"
        - name: node-2
          ip: "192.168.56.11"

🟢 03 - Cluster installation (assistant)

Vagrantfile

Vagrant.configure("2") do |config|

    config.vm.provider "virtualbox" do |vb|
        vb.memory = "4096"
        vb.cpus = "4"
    end

    config.vm.define "node1" do |node1|
        node1.vm.box = "generic/rhel7"
        node1.vm.synced_folder ".", "/vagrant"
        node1.vm.network "private_network", ip: "192.168.56.10"

        node1.vm.provision "shell", inline: <<-SHELL
            # Disable firewall
            systemctl stop firewalld
            systemctl disable firewalld
        SHELL
    end

    config.vm.define "node2" do |node2|
        node2.vm.box = "generic/rhel7"
        node2.vm.synced_folder ".", "/vagrant"
        node2.vm.network "private_network", ip: "192.168.56.11"

        node2.vm.provision "shell", inline: <<-SHELL
            # Disable firewall
            systemctl stop firewalld
            systemctl disable firewalld
        SHELL
    end
end

  1. Create wazuh-install.sh from the 4.8.0 branch

    Details

    git checkout 4.8.0
    bash unattended_installer/builder.sh -i -d staging

  2. Generate config files

    Details

     bash wazuh-install.sh --generate-config-files
    10/11/2023 13:44:02 INFO: Starting Wazuh installation assistant. Wazuh version: 4.8.0
    10/11/2023 13:44:02 INFO: Verbose logging redirected to /var/log/wazuh-install.log
    10/11/2023 13:44:14 INFO: Verifying that your system meets the recommended minimum hardware requirements.
    10/11/2023 13:44:14 INFO: --- Configuration files ---
    10/11/2023 13:44:14 INFO: Generating configuration files.
    10/11/2023 13:44:15 INFO: Created wazuh-install-files.tar. It contains the Wazuh cluster key, certificates, and passwords necessary for installation.

  3. Install indexer node-1

    Details

    bash wazuh-install.sh --wazuh-indexer node-1
    10/11/2023 13:47:33 INFO: Starting Wazuh installation assistant. Wazuh version: 4.8.0
    10/11/2023 13:47:33 INFO: Verbose logging redirected to /var/log/wazuh-install.log
    10/11/2023 13:47:44 INFO: Verifying that your system meets the recommended minimum hardware requirements.
    10/11/2023 13:47:47 INFO: Wazuh development repository added.
    10/11/2023 13:47:48 INFO: --- Wazuh indexer ---
    10/11/2023 13:47:48 INFO: Starting Wazuh indexer installation.
    10/11/2023 13:50:45 INFO: Wazuh indexer installation finished.
    10/11/2023 13:50:45 INFO: Wazuh indexer post-install configuration finished.
    10/11/2023 13:50:45 INFO: Starting service wazuh-indexer.
    10/11/2023 13:51:29 INFO: wazuh-indexer service started.
    10/11/2023 13:51:29 INFO: Initializing Wazuh indexer cluster security settings.
    10/11/2023 13:51:29 INFO: Wazuh indexer cluster initialized.
    10/11/2023 13:51:29 INFO: Installation finished.

  4. Install indexer node-2

    Details

    bash wazuh-install.sh --wazuh-indexer node-2
    10/11/2023 13:49:48 INFO: Starting Wazuh installation assistant. Wazuh version: 4.8.0
    10/11/2023 13:49:48 INFO: Verbose logging redirected to /var/log/wazuh-install.log
    10/11/2023 13:50:00 INFO: Verifying that your system meets the recommended minimum hardware requirements.
    10/11/2023 13:50:03 INFO: Wazuh development repository added.
    10/11/2023 13:50:04 INFO: --- Wazuh indexer ---
    10/11/2023 13:50:04 INFO: Starting Wazuh indexer installation.
    10/11/2023 13:53:10 INFO: Wazuh indexer installation finished.
    10/11/2023 13:53:10 INFO: Wazuh indexer post-install configuration finished.
    10/11/2023 13:53:10 INFO: Starting service wazuh-indexer.
    10/11/2023 13:53:24 INFO: wazuh-indexer service started.
    10/11/2023 13:53:24 INFO: Initializing Wazuh indexer cluster security settings.
    10/11/2023 13:53:24 INFO: Wazuh indexer cluster initialized.
    10/11/2023 13:53:24 INFO: Installation finished.

  5. Initialize cluster

    Details

    bash wazuh-install.sh --start-cluster
    10/11/2023 14:04:27 INFO: Starting Wazuh installation assistant. Wazuh version: 4.8.0
    10/11/2023 14:04:27 INFO: Verbose logging redirected to /var/log/wazuh-install.log
    10/11/2023 14:04:39 INFO: Verifying that your system meets the recommended minimum hardware requirements.
    10/11/2023 14:04:51 INFO: Wazuh indexer cluster security configuration initialized.
    10/11/2023 14:04:53 INFO: The Wazuh indexer cluster ISM initialized.
    10/11/2023 14:05:55 INFO: Updating the internal users.
    10/11/2023 14:05:57 INFO: A backup of the internal users has been saved in the /etc/wazuh-indexer/internalusers-backup folder.
    10/11/2023 14:06:06 INFO: Wazuh indexer cluster started.

  6. Check rollover policy

    Details

    curl -X GET https://192.168.56.11:9200/_plugins/_ism/explain?pretty -k -u admin:xA*XezzKolMmhRBofw?*kOFA.Frr7BGt
    {
      "wazuh-alerts-4.x-2023.11.10-000001" : {
        "index.plugins.index_state_management.policy_id" : "rollover_policy",
        "index.opendistro.index_state_management.policy_id" : "rollover_policy",
        "index" : "wazuh-alerts-4.x-2023.11.10-000001",
        "index_uuid" : "aZwswJFxQI2kQT5c_p55sw",
        "policy_id" : "rollover_policy",
        "policy_seq_no" : -2,
        "policy_primary_term" : 0,
        "rolled_over" : false,
        "index_creation_date" : 1699625093269,
        "state" : {
          "name" : "active",
          "start_time" : 1699625441488
        },
        "action" : {
          "name" : "rollover",
          "start_time" : 1699625783263,
          "index" : 0,
          "failed" : false,
          "consumed_retries" : 0,
          "last_retry_time" : 0
        },
        "step" : {
          "name" : "attempt_rollover",
          "start_time" : 1699625783265,
          "step_status" : "condition_not_met"
        },
        "retry_info" : {
          "failed" : false,
          "consumed_retries" : 0
        },
        "info" : {
          "message" : "Pending rollover of index [index=wazuh-alerts-4.x-2023.11.10-000001]",
          "conditions" : {
            "min_primary_shard_size" : {
              "condition" : "25gb",
              "current" : "0b",
              "shard" : 0
            },
            "min_index_age" : {
              "condition" : "7d",
              "current" : "57.5m",
              "creationDate" : 1699625093269
            },
            "min_doc_count" : {
              "condition" : 200000000,
              "current" : 0
            }
          }
        },
        "enabled" : true
      },
      "wazuh-archives-4.x-2023.11.10-000001" : {
        "index.plugins.index_state_management.policy_id" : "rollover_policy",
        "index.opendistro.index_state_management.policy_id" : "rollover_policy",
        "index" : "wazuh-archives-4.x-2023.11.10-000001",
        "index_uuid" : "O9VRPfz8TwaROAO-tB6pDw",
        "policy_id" : "rollover_policy",
        "policy_seq_no" : -2,
        "policy_primary_term" : 0,
        "rolled_over" : false,
        "index_creation_date" : 1699625093593,
        "state" : {
          "name" : "active",
          "start_time" : 1699625427540
        },
        "action" : {
          "name" : "rollover",
          "start_time" : 1699625866044,
          "index" : 0,
          "failed" : false,
          "consumed_retries" : 0,
          "last_retry_time" : 0
        },
        "step" : {
          "name" : "attempt_rollover",
          "start_time" : 1699625866044,
          "step_status" : "condition_not_met"
        },
        "retry_info" : {
          "failed" : false,
          "consumed_retries" : 0
        },
        "info" : {
          "message" : "Pending rollover of index [index=wazuh-archives-4.x-2023.11.10-000001]",
          "conditions" : {
            "min_primary_shard_size" : {
              "condition" : "25gb",
              "current" : "0b",
              "shard" : 0
            },
            "min_index_age" : {
              "condition" : "7d",
              "current" : "55m",
              "creationDate" : 1699625093593
            },
            "min_doc_count" : {
              "condition" : 200000000,
              "current" : 0
            }
          }
        },
        "enabled" : true
      },
      "total_managed_indices" : 2
    }

🟢 04 - Cluster installation (step-by-step)

I've used the SSL certificates generated by the assistant.

  1. Create rpm package from the 4.8.0 branch

    Details

    bash stack/indexer/rpm/build_package.sh -s /tmp

  2. Install package: indexer node-1

    Details

    yum localinstall wazuh-indexer-4.8.0-1.x86_64.rpm

  3. Install package: indexer node-2

    Details

    yum localinstall wazuh-indexer-4.8.0-1.x86_64.rpm

  4. Configuring the nodes and deploying the certificates

    Details

    • Edit the opensearch.yml accordingly
    network.host: "192.168.56.10"
    node.name: "node-1"
    cluster.initial_master_nodes:
        - "node-1"
        - "node-2"
    cluster.name: "wazuh-cluster"
    discovery.seed_hosts:
        - "192.168.56.10"
        - "192.168.56.11"
    node.max_local_storage_nodes: "3"
    path.data: /var/lib/wazuh-indexer
    path.logs: /var/log/wazuh-indexer
    
    plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/node-1.pem
    plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/node-1-key.pem
    plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
    plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/node-1.pem
    plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/node-1-key.pem
    plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
    plugins.security.ssl.http.enabled: true
    plugins.security.ssl.transport.enforce_hostname_verification: false
    plugins.security.ssl.transport.resolve_hostname: false
    
    plugins.security.authcz.admin_dn:
        - "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
    plugins.security.check_snapshot_restore_write_privileges: true
    plugins.security.enable_snapshot_restore_privilege: true
    plugins.security.nodes_dn:
        - "CN=node-1,OU=Wazuh,O=Wazuh,L=California,C=US"
        - "CN=node-2,OU=Wazuh,O=Wazuh,L=California,C=US"
    plugins.security.restapi.roles_enabled:
        - "all_access"
        - "security_rest_api_access"
    
    plugins.security.system_indices.enabled: true
    plugins.security.system_indices.indices:
        [
            ".plugins-ml-model",
            ".plugins-ml-task",
            ".opendistro-alerting-config",
            ".opendistro-alerting-alert*",
            ".opendistro-anomaly-results*",
            ".opendistro-anomaly-detector*",
            ".opendistro-anomaly-checkpoints",
            ".opendistro-anomaly-detection-state",
            ".opendistro-reports-*",
            ".opensearch-notifications-*",
            ".opensearch-notebooks",
            ".opensearch-observability",
            ".opendistro-asynchronous-search-response*",
            ".replication-metadata-store",
        ]
    
    ### Option to allow Filebeat-oss 7.10.2 to work ###
    compatibility.override_main_response_version: true
    • Move the certificates to the right place
    mkdir /etc/wazuh-indexer/certs
    cp wazuh-install-files/* /etc/wazuh-indexer/certs
    chmod 500 /etc/wazuh-indexer/certs
    chmod 400 /etc/wazuh-indexer/certs/*
    chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs

  5. Initialize the cluster

    Details

    • Start the node
    systemctl daemon-reload
    systemctl enable wazuh-indexer
    systemctl start wazuh-indexer
    • Initialize the cluster
    bash /usr/share/wazuh-indexer/bin/indexer-init.sh -i 192.168.56.10
    Executing Wazuh indexer security init script...
    **************************************************************************
    ** This tool will be deprecated in the next major release of OpenSearch **
    ** https://github.com/opensearch-project/security/issues/1755           **
    **************************************************************************
    Security Admin v7
    Will connect to 192.168.56.11:9200 ... done
    Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
    OpenSearch Version: 2.10.0
    Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
    Clustername: wazuh-cluster
    Clusterstate: GREEN
    Number of nodes: 2
    Number of data nodes: 2
    .opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
    Populate config from /etc/wazuh-indexer/opensearch-security/
    Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml
      SUCC: Configuration for 'config' created or updated
    Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml
      SUCC: Configuration for 'roles' created or updated
    Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml
      SUCC: Configuration for 'rolesmapping' created or updated
    Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml
      SUCC: Configuration for 'internalusers' created or updated
    Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml
      SUCC: Configuration for 'actiongroups' created or updated
    Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml
      SUCC: Configuration for 'tenants' created or updated
    Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml
      SUCC: Configuration for 'nodesdn' created or updated
    Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml
      SUCC: Configuration for 'whitelist' created or updated
    Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml
      SUCC: Configuration for 'audit' created or updated
    Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml
      SUCC: Configuration for 'allowlist' created or updated
    SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
    SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
    Done with success
    Executing Wazuh indexer ISM init script...
    wazuh-alerts template uploaded
    wazuh-archives template uploaded
    rollover_policy policy uploaded
    wazuh-alerts write index created
    wazuh-archives write index created
    Indexer ISM initialization finished successfully

  6. Check rollover policy

    Details

    curl -X GET https://192.168.56.11:9200/_plugins/_ism/explain?pretty -k -u admin:admin
    {
      "wazuh-alerts-4.x-2023.11.10-000001" : {
        "index.plugins.index_state_management.policy_id" : "rollover_policy",
        "index.opendistro.index_state_management.policy_id" : "rollover_policy",
        "index" : "wazuh-alerts-4.x-2023.11.10-000001",
        "index_uuid" : "xbk7-diOTQiyTpkPfjwqsA",
        "policy_id" : "rollover_policy",
        "enabled" : true
      },
      "wazuh-archives-4.x-2023.11.10-000001" : {
        "index.plugins.index_state_management.policy_id" : "rollover_policy",
        "index.opendistro.index_state_management.policy_id" : "rollover_policy",
        "index" : "wazuh-archives-4.x-2023.11.10-000001",
        "index_uuid" : "20BAoHYfRjmlk84TsQ85fA",
        "policy_id" : "rollover_policy",
        "enabled" : true
      },
      "total_managed_indices" : 2
    }

🟢 06 - Cluster upgrade without existing rollover policy

  1. Install wazuh-indexer 4.6.0 using the assistant

    Vagrantfile

    Vagrant.configure("2") do |config|
    
      config.vm.provider "virtualbox" do |vb|
          vb.memory = "4096"
          vb.cpus = "4"
      end
    
      config.vm.define "node1" do |node1|
          node1.vm.box = "generic/rhel7"
          node1.vm.synced_folder ".", "/vagrant"
          node1.vm.network "private_network", ip: "192.168.56.10"
    
          node1.vm.provision "shell", inline: <<-SHELL
              # Disable firewall
              systemctl stop firewalld
              systemctl disable firewalld
    
              cd /vagrant
              curl -sO https://packages.wazuh.com/4.6/wazuh-install.sh
              bash wazuh-install.sh --wazuh-indexer node-1
          SHELL
      end
    
      config.vm.define "node2" do |node2|
          node2.vm.box = "generic/rhel7"
          node2.vm.synced_folder ".", "/vagrant"
          node2.vm.network "private_network", ip: "192.168.56.11"
    
          node2.vm.provision "shell", inline: <<-SHELL
              # Disable firewall
              systemctl stop firewalld
              systemctl disable firewalld
    
              cd /vagrant
              curl -sO https://packages.wazuh.com/4.6/wazuh-install.sh
              bash wazuh-install.sh --wazuh-indexer node-2
          SHELL
      end
    end

    Cluster initialized with the following command:

    bash wazuh-install.sh --start-cluster

  2. Upgrade wazuh-indexer using local RPM package generated from 4.8.0 branch

    Details

    systemctl stop wazuh-indexer
    yum localinstall /vagrant/wazuh-indexer-4.8.0-1.x86_64.rpm

  3. Restart the service

    Details

    systemctl daemon-reload
    systemctl enable wazuh-indexer
    systemctl start wazuh-indexer

  4. Re-initialize the cluster

    Details

    bash /usr/share/wazuh-indexer/bin/indexer-init.sh -i 192.165.56.10 -p admin
    
    Executing Wazuh indexer security init script...
    **************************************************************************
    ** This tool will be deprecated in the next major release of OpenSearch **
    ** https://github.com/opensearch-project/security/issues/1755           **
    **************************************************************************
    Security Admin v7
    Will connect to 192.168.56.11:9200 ... done
    Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
    OpenSearch Version: 2.10.0
    Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
    Clustername: wazuh-indexer-cluster
    Clusterstate: GREEN
    Number of nodes: 2
    Number of data nodes: 2
    .opendistro_security index already exists, so we do not need to create one.
    Populate config from /etc/wazuh-indexer/opensearch-security/
    Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml
      SUCC: Configuration for 'config' created or updated
    Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml
      SUCC: Configuration for 'roles' created or updated
    Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml
      SUCC: Configuration for 'rolesmapping' created or updated
    Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml
      SUCC: Configuration for 'internalusers' created or updated
    Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml
      SUCC: Configuration for 'actiongroups' created or updated
    Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml
      SUCC: Configuration for 'tenants' created or updated
    Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml
      SUCC: Configuration for 'nodesdn' created or updated
    Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml
      SUCC: Configuration for 'whitelist' created or updated
    Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml
      SUCC: Configuration for 'audit' created or updated
    Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml
      SUCC: Configuration for 'allowlist' created or updated
    SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
    SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
    Done with success
    Executing Wazuh indexer ISM init script...
    wazuh-alerts template uploaded
    wazuh-archives template uploaded
    rollover_policy policy uploaded
    wazuh-alerts write index created
    wazuh-archives write index created
    Indexer ISM initialization finished successfully
  5. Check rollover policy

    Details

    curl -k -u admin:admin https://192.168.56.10:9200/_plugins/_ism/explain?pretty
    {
      "wazuh-alerts-4.x-2023.11.10-000001" : {
        "index.plugins.index_state_management.policy_id" : "rollover_policy",
        "index.opendistro.index_state_management.policy_id" : "rollover_policy",
        "index" : "wazuh-alerts-4.x-2023.11.10-000001",
        "index_uuid" : "oQZxcCllTpyOBI8bENxmSw",
        "policy_id" : "rollover_policy",
        "enabled" : true
      },
      "wazuh-archives-4.x-2023.11.10-000001" : {
        "index.plugins.index_state_management.policy_id" : "rollover_policy",
        "index.opendistro.index_state_management.policy_id" : "rollover_policy",
        "index" : "wazuh-archives-4.x-2023.11.10-000001",
        "index_uuid" : "bZP_f2iITTO3u5lr2On6qQ",
        "policy_id" : "rollover_policy",
        "enabled" : true
      },
      "total_managed_indices" : 2
    }

🔴 08 - Cluster upgrade with existing rollover policy

  1. Install wazuh-indexer 4.6.0 using the assistant

    Vagrantfile

    Same as on test 06.

  2. Create a rollover policy

    Details

    C_AUTH="-u admin:wxefR540b3bm?SPRezZ6jGOw7ViqnNb6"
    INDEXER_URL="https://192.168.56.10:9200"
    POLICY_NAME="test_policy"
    
    curl -k ${C_AUTH} -X PUT "${INDEXER_URL}/_plugins/_ism/policies/${POLICY_NAME}" \
        -H 'Content-Type: application/json' \
        -d '
        {
            "policy": {
                "description": "Wazuh rollover and alias policy",
                "default_state": "active",
                "states": [
                    {
                        "name": "active",
                        "actions": [
                            {
                                "rollover": {
                                    "min_primary_shard_size": "10gb",
                                    "min_index_age": "1d",
                                    "min_doc_count": "500"
                                }
                            }
                        ]
                    }
                ],
                "ism_template": {
                    "index_patterns": ["wazuh-alerts-*", "wazuh-archives-*", "-wazuh-alerts-4.x-sample*"],
                    "priority": "50"
                }
            }
        }
    '

  3. Upgrade wazuh-indexer using local RPM package generated from 4.8.0 branch

    Details

    systemctl stop wazuh-indexer
    yum localinstall /vagrant/wazuh-indexer-4.8.0-1.x86_64.rpm
    
    Failed to set locale, defaulting to C
    Loaded plugins: product-id, search-disabled-repos
    Examining /vagrant/wazuh-indexer-4.8.0-1.x86_64.rpm: wazuh-indexer-4.8.0-1.x86_64
    Marking /vagrant/wazuh-indexer-4.8.0-1.x86_64.rpm as an update to wazuh-indexer-4.6.0-1.x86_64
    Resolving Dependencies
    --> Running transaction check
    ---> Package wazuh-indexer.x86_64 0:4.6.0-1 will be updated
    ---> Package wazuh-indexer.x86_64 0:4.8.0-1 will be an update
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    =====================================================================================================================
    Package                   Arch               Version                Repository                                 Size
    =====================================================================================================================
    Updating:
    wazuh-indexer             x86_64             4.8.0-1                /wazuh-indexer-4.8.0-1.x86_64             1.0 G
    
    Transaction Summary
    =====================================================================================================================
    Upgrade  1 Package
    
    Total size: 1.0 G
    Is this ok [y/d/N]: y
    Downloading packages:
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Updating   : wazuh-indexer-4.8.0-1.x86_64                                                                      1/2 
      Cleanup    : wazuh-indexer-4.6.0-1.x86_64                                                                      2/2 
      Verifying  : wazuh-indexer-4.8.0-1.x86_64                                                                      1/2 
      Verifying  : wazuh-indexer-4.6.0-1.x86_64                                                                      2/2 
    
    Updated:
      wazuh-indexer.x86_64 0:4.8.0-1                                                                                     
    
    Complete!

  4. Restart the service

    Details

    systemctl daemon-reload
    systemctl enable wazuh-indexer
    systemctl start wazuh-indexer
    curl -k ${C_AUTH} -X GET "${INDEXER_URL}/_cat/nodes?v"
    ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles                               cluster_manager name
    192.168.56.11           10          97  29    0.52    0.38     0.22 dimr      data,ingest,master,remote_cluster_client -               node-2
    192.168.56.10           27          97  29    0.63    0.38     0.20 dimr      data,ingest,master,remote_cluster_client *               node-1

  5. Re-initialize the cluster 🔴

    Details

    1. 🟢 Scenario A: lower prio
      bash /usr/share/wazuh-indexer/bin/indexer-init.sh -i 192.168.56.10 -p +YO57pQKHvrG84qg5kFI9VNlD+Qqmmi3 -P 2
      Executing Wazuh indexer security init script...
      **************************************************************************
      ** This tool will be deprecated in the next major release of OpenSearch **
      ** https://github.com/opensearch-project/security/issues/1755           **
      **************************************************************************
      Security Admin v7
      Will connect to 192.168.56.10:9200 ... done
      Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
      OpenSearch Version: 2.10.0
      Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
      Clustername: wazuh-indexer-cluster
      Clusterstate: GREEN
      Number of nodes: 2
      Number of data nodes: 2
      .opendistro_security index already exists, so we do not need to create one.
      Populate config from /etc/wazuh-indexer/opensearch-security/
      Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml 
        SUCC: Configuration for 'config' created or updated
      Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml 
        SUCC: Configuration for 'roles' created or updated
      Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml 
        SUCC: Configuration for 'rolesmapping' created or updated
      Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml 
        SUCC: Configuration for 'internalusers' created or updated
      Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml 
        SUCC: Configuration for 'actiongroups' created or updated
      Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml 
        SUCC: Configuration for 'tenants' created or updated
      Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml 
        SUCC: Configuration for 'nodesdn' created or updated
      Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml 
        SUCC: Configuration for 'whitelist' created or updated
      Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml 
        SUCC: Configuration for 'audit' created or updated
      Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml 
        SUCC: Configuration for 'allowlist' created or updated
      SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
      SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
      Done with success
      Executing Wazuh indexer ISM init script...
      wazuh-alerts template uploaded
      wazuh-archives template uploaded
      rollover_policy policy uploaded
      wazuh-alerts write index created
      wazuh-archives write index created
      Indexer ISM initialization finished successfully
    • Check which policy is created:

      curl -k ${C_AUTH} -X GET "${INDEXER_URL}/_plugins/_ism/policies/${POLICY_NAME}?pretty"
      {
        "_id" : "rollover_policy",
        "_version" : 1,
        "_seq_no" : 65,
        "_primary_term" : 2,
        "policy" : {
          "policy_id" : "rollover_policy",
          "description" : "Wazuh rollover and alias policy",
          "last_updated_time" : 1699987034225,
          "schema_version" : 19,
          "error_notification" : null,
          "default_state" : "active",
          "states" : [
            {
              "name" : "active",
              "actions" : [
                {
                  "retry" : {
                    "count" : 3,
                    "backoff" : "exponential",
                    "delay" : "1m"
                  },
                  "rollover" : {
                    "min_doc_count" : 200000000,
                    "min_index_age" : "7d",
                    "min_primary_shard_size" : "25gb",
                    "copy_alias" : false
                  }
                }
              ],
              "transitions" : [ ]
            }
          ],
          "ism_template" : [
            {
              "index_patterns" : [
                "wazuh-alerts-*",
                "wazuh-archives-*",
                "-wazuh-alerts-4.x-sample*"
              ],
              "priority" : 2,
              "last_updated_time" : 1699987034225
            }
          ]
        }
      }
      
    • Check that the existing policy takes precedence:

      curl -k ${C_AUTH} "${INDEXER_URL}/_plugins/_ism/explain?pretty"
      {
        "wazuh-alerts-4.x-2023.11.14-000001" : {
          "index.plugins.index_state_management.policy_id" : "test_policy",
          "index.opendistro.index_state_management.policy_id" : "test_policy",
          "index" : "wazuh-alerts-4.x-2023.11.14-000001",
          "index_uuid" : "22B_E0M5RWiofkbX3l02yg",
          "policy_id" : "test_policy",
          "policy_seq_no" : -2,
          "policy_primary_term" : 0,
          "rolled_over" : false,
          "index_creation_date" : 1699983851534,
          "state" : {
            "name" : "active",
            "start_time" : 1699984290317
          },
          "action" : {
            "name" : "rollover",
            "start_time" : 1699984572713,
            "index" : 0,
            "failed" : false,
            "consumed_retries" : 0,
            "last_retry_time" : 0
          },
          "step" : {
            "name" : "attempt_rollover",
            "start_time" : 1699984572713,
            "step_status" : "condition_not_met"
          },
          "retry_info" : {
            "failed" : false,
            "consumed_retries" : 0
          },
          "info" : {
            "message" : "Pending rollover of index [index=wazuh-alerts-4.x-2023.11.14-000001]",
            "conditions" : {
              "min_primary_shard_size" : {
                "condition" : "10gb",
                "current" : "0b",
                "shard" : 0
              },
              "min_index_age" : {
                "condition" : "1d",
                "current" : "1h",
                "creationDate" : 1699983851534
              },
              "min_doc_count" : {
                "condition" : 500,
                "current" : 0
              }
            }
          },
          "enabled" : true
        },
        "wazuh-archives-4.x-2023.11.14-000001" : {
          "index.plugins.index_state_management.policy_id" : "test_policy",
          "index.opendistro.index_state_management.policy_id" : "test_policy",
          "index" : "wazuh-archives-4.x-2023.11.14-000001",
          "index_uuid" : "rRPowrOqSSymMWig2FEueg",
          "policy_id" : "test_policy",
          "policy_seq_no" : -2,
          "policy_primary_term" : 0,
          "rolled_over" : false,
          "index_creation_date" : 1699983851836,
          "state" : {
            "name" : "active",
            "start_time" : 1699984310913
          },
          "action" : {
            "name" : "rollover",
            "start_time" : 1699984455645,
            "index" : 0,
            "failed" : false,
            "consumed_retries" : 0,
            "last_retry_time" : 0
          },
          "step" : {
            "name" : "attempt_rollover",
            "start_time" : 1699984455646,
            "step_status" : "condition_not_met"
          },
          "retry_info" : {
            "failed" : false,
            "consumed_retries" : 0
          },
          "info" : {
            "message" : "Pending rollover of index [index=wazuh-archives-4.x-2023.11.14-000001]",
            "conditions" : {
              "min_primary_shard_size" : {
                "condition" : "10gb",
                "current" : "0b",
                "shard" : 0
              },
              "min_index_age" : {
                "condition" : "1d",
                "current" : "1h",
                "creationDate" : 1699983851836
              },
              "min_doc_count" : {
                "condition" : 500,
                "current" : 0
              }
            }
          },
          "enabled" : true
        },
        "total_managed_indices" : 2
      }
    1. 🔴 Scenario B: equal prio
    • Delete the rollover_policy and add it back with a custom priority:

      curl -k ${C_AUTH} -X DELETE "${INDEXER_URL}/_plugins/_ism/policies/${POLICY_NAME}?pretty"
      curl -k ${C_AUTH} -X DELETE "${INDEXER_URL}/wazuh-a*-000001"
      curl -k ${C_AUTH} -X DELETE "${INDEXER_URL}/_template/wazuh-a*"
      bash /usr/share/wazuh-indexer/bin/indexer-ism-init.sh -i 192.168.56.10 -p +YO57pQKHvrG84qg5kFI9VNlD+Qqmmi3 -P 50
      
      
      Executing Wazuh indexer ISM init script...
      wazuh-alerts template uploaded
      wazuh-archives template uploaded
      rollover_policy policy uploaded
      wazuh-alerts write index created
      wazuh-archives write index created
      Indexer ISM initialization finished successfully
    • Check which policy is created:

      curl -k ${C_AUTH} -X GET "${INDEXER_URL}/_plugins/_ism/policies/${POLICY_NAME}?pretty"
      
      {
        "error" : {
          "root_cause" : [
            {
              "type" : "status_exception",
              "reason" : "Policy not found"
            }
          ],
          "type" : "status_exception",
          "reason" : "Policy not found"
        },
        "status" : 404
      }

      Debugging why it wasn't created, I got this:

      [root@rhel7 vagrant]# echo $?
      0

      The policy isn't uploaded but cURL returns 0, so the validation passes and the script continues.
      I'll open an issue to fix this. We should validate the request using the HTTP status code
      instead of the exit code of the cURL command.

    • Check which policy takes precedence: skipped

    1. 🔴 Scenario C: higher prio
    • Delete the rollover_policy and add it back with a custom priority:

        curl -k ${C_AUTH} -X DELETE "${INDEXER_URL}/_plugins/_ism/policies/${POLICY_NAME}?pretty"
        curl -k ${C_AUTH} -X DELETE "${INDEXER_URL}/wazuh-a*-000001"
        curl -k ${C_AUTH} -X DELETE "${INDEXER_URL}/_template/wazuh-a*"
        bash /usr/share/wazuh-indexer/bin/indexer-ism-init.sh -i 192.168.56.10 -p +YO57pQKHvrG84qg5kFI9VNlD+Qqmmi3 -P 100
      
        wazuh-alerts template uploaded
        wazuh-archives template uploaded
        rollover_policy policy uploaded
        Indexer ISM initialization finished successfully
    • Check which policy is created:

      curl -k ${C_AUTH} -X GET "${INDEXER_URL}/_plugins/_ism/policies/${POLICY_NAME}?pretty" 
      {
        "_id" : "rollover_policy",
        "_version" : 3,
        "_seq_no" : 93,
        "_primary_term" : 2,
        "policy" : {
          "policy_id" : "rollover_policy",
          "description" : "Wazuh rollover and alias policy",
          "last_updated_time" : 1699987934033,
          "schema_version" : 19,
          "error_notification" : null,
          "default_state" : "active",
          "states" : [
            {
              "name" : "active",
              "actions" : [
                {
                  "retry" : {
                    "count" : 3,
                    "backoff" : "exponential",
                    "delay" : "1m"
                  },
                  "rollover" : {
                    "min_doc_count" : 200000000,
                    "min_index_age" : "7d",
                    "min_primary_shard_size" : "25gb",
                    "copy_alias" : false
                  }
                }
              ],
              "transitions" : [ ]
            }
          ],
          "ism_template" : [
            {
              "index_patterns" : [
                "wazuh-alerts-*",
                "wazuh-archives-*",
                "-wazuh-alerts-4.x-sample*"
              ],
              "priority" : 100,
              "last_updated_time" : 1699987934033
            }
          ]
        }
      }
    • Check that the existing policy takes precedence:

      curl -k ${C_AUTH} "${INDEXER_URL}/_plugins/_ism/explain?pretty"
      {
        "wazuh-alerts-4.x-2023.11.14-000001" : {
          "index.plugins.index_state_management.policy_id" : "test_policy",
          "index.opendistro.index_state_management.policy_id" : "test_policy",
          "index" : "wazuh-alerts-4.x-2023.11.14-000001",
          "index_uuid" : "e6mSRdlgTs6SsveMzJEraw",
          "policy_id" : "test_policy",
          "enabled" : true
        },
        "wazuh-archives-4.x-2023.11.14-000001" : {
          "index.plugins.index_state_management.policy_id" : "test_policy",
          "index.opendistro.index_state_management.policy_id" : "test_policy",
          "index" : "wazuh-archives-4.x-2023.11.14-000001",
          "index_uuid" : "FW6BqCPqSWub_mf4onrmfw",
          "policy_id" : "test_policy",
          "enabled" : true
        },
        "total_managed_indices" : 2
      }

      The previous policy is still in use until the next rollover, which we can force with _rollover API:

      curl -k ${C_AUTH} -X POST "${INDEXER_URL}/wazuh-alerts/_rollover?pretty" 
      {
        "acknowledged" : true,
        "shards_acknowledged" : true,
        "old_index" : "wazuh-alerts-4.x-2023.11.14-000001",
        "new_index" : "wazuh-alerts-4.x-2023.11.14-000002",
        "rolled_over" : true,
        "dry_run" : false,
        "conditions" : { }
      }

      We check the indices again:

      curl -k ${C_AUTH} "${INDEXER_URL}/_plugins/_ism/explain?pretty"
      {
        "wazuh-alerts-4.x-2023.11.14-000001" : {
          "index.plugins.index_state_management.policy_id" : "test_policy",
          "index.opendistro.index_state_management.policy_id" : "test_policy",
          "index" : "wazuh-alerts-4.x-2023.11.14-000001",
          "index_uuid" : "e6mSRdlgTs6SsveMzJEraw",
          "policy_id" : "test_policy",
          "enabled" : true
        },
        "wazuh-alerts-4.x-2023.11.14-000002" : {
          "index.plugins.index_state_management.policy_id" : "rollover_policy",
          "index.opendistro.index_state_management.policy_id" : "rollover_policy",
          "index" : "wazuh-alerts-4.x-2023.11.14-000002",
          "index_uuid" : "U_YejLFfSxCfiNM05mNuHA",
          "policy_id" : "rollover_policy",
          "enabled" : true
        },
        "wazuh-archives-4.x-2023.11.14-000001" : {
          "index.plugins.index_state_management.policy_id" : "test_policy",
          "index.opendistro.index_state_management.policy_id" : "test_policy",
          "index" : "wazuh-archives-4.x-2023.11.14-000001",
          "index_uuid" : "FW6BqCPqSWub_mf4onrmfw",
          "policy_id" : "test_policy",
          "enabled" : true
        },
        "total_managed_indices" : 3
      }

      The rollover_policy is now in use.

      We need to document this behaviour.

@gdiazlo gdiazlo added request/operational Operational requests and removed request/operational Operational requests labels Nov 13, 2023
@AlexRuiz7
Copy link
Member Author

AlexRuiz7 commented Nov 14, 2023

Issues created

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
level/task Task issue request/operational Operational requests type/test Test issue
Projects
No open projects
Archived in project
Development

No branches or pull requests

2 participants