System test: New design research

[Rebits]

Description

This issue aims to report on the current status of system testing and discuss the new design. Some approaches will be proposed.

TODO

• Research current design: System test: New design research #2440 (comment)
• Brainstormong ad proposed new system test designs
• Meeting and selection of new model

[roronoasins]

Current design

Deployment

Currently, to run the system tests we deploy docker containers with ansible connection. You can deploy and destroy a few of environment types.

Deployment sample

Navigating to the desired environment path, you can deploy and provision it by running:

ansible-playbook -i inventory.yml playbook.yml --extra-vars='{"wazuh_branch":"master"}'

For basic_cluster we need to specify also the qa_branch:

ansible-playbook -i inventory.yml playbook.yml --extra-vars='{"wazuh_branch":"master", "wazuh_qa_branch":"master"}'`

Destroy sample

To destroy the environment, you must be in the desired environment path and run:

ansible-playbook -i inventory.yml destroy.yml

Provisioning

With ansible(playbook, inventory) we can deploy a few types of environments(both deployment and provisioning are performed together):

agentless_cluster

• wazuh-master
• wazuh-worker1
• wazuh-worker2

basic_cluster

• wazuh-master
• wazuh-worker1
• wazuh-worker2
• wazuh-agent1
• wazuh-agent2
• wazuh-agent3

basic_environment

• wazuh-master
• wazuh-agent1

enrollment_cluster

• wazuh-master
• wazuh-worker1
• wazuh-worker2
• wazuh-agent1

manager_agent

• wazuh-master
• wazuh-agent1
• wazuh-agent2
• wazuh-agent3

one_manager_agent

• wazuh-master
• wazuh-agent1

---

These provisioning types have the following files:

• destroy.yml
• inventory.yml
• playbook.yml

Where most of them are equal(e.g, when creating wazuh-worker2 with playbook.yml files or destroying some component)
Also the ansible.cfg is equal.

Framework components

HostManager

Ansible wrapper that is used to modify/run/control the remote host via ansible python api.

HostMonitor

Monitors remote files to verify that expected events are taking place. This class has a path to a file which has the callbacks and timeouts

wazuh-worker2:
  - regex: '.*Received.*shared files to update from .*'
    path: "/var/ossec/logs/cluster.log"
    timeout: 60
  - regex: '.*Worker wazuh-worker2.*Integrity sync.*Finished in.*'
    path: "/var/ossec/logs/cluster.log"
    timeout: 60
  - regex: '.*Worker wazuh-worker2.*Integrity sync.*Starting.*'
    path: "/var/ossec/logs/cluster.log"
    timeout: 60
  - regex: '.*Worker wazuh-worker2.*Integrity sync.*Finished in.*'
    path: "/var/ossec/logs/cluster.log"
    timeout: 60

wazuh-master:
  - regex: '.*Worker wazuh-worker2.*Integrity check.*Finished in.*'
    path: "/var/ossec/logs/cluster.log"
    timeout: 60

For every path within each host a process is created, monitoring that file using a regex.

Threads are created and allocated in _file_monitors(file_composer) and _file_content_collectors(_start). For each desired file, the file content is dumped into the output_path file. Then, it monitors that output_path using a QueueMonitor. Finally, it collects the results that have been inserted into the queue.

When a thread ends, the result is added to the result dictionary with the host key. If the message is a TimeoutError it performs a raise(only reporting one error if there are more errors just the first one shows up).

Pros/Cons

Deployment

• Pros

  • Docker containers deployment is so fast
  • Both local and jenkins automation develop

• Cons

  • It has system limitation, only debian can be used
  • Deployment and provisioning are performed at the same time
  • For every deploy you must navigate to its path
  • If we have a wrong environment, we have to destroy it and deploy it again
  • Differents environments types cannot be deployed at the same time

Provisioning

• Pros

  • Unattended

• Cons

  • It compiles the source code instead of use a package
  • Duplicated snippets -> playbooks, inventory, etc.
  • Once deployed and provisioned, you cannot run with another wazuh(-qa) branch

Testing

• Pros

  • Allows running system test so easy

• Cons

  • Each test type is different
  • Most of them, has no setup/teardown
  • Unstable framework
  • A report when having errors its needed
  • Tests do not support all systems
  • We cannot run different system combinations

[Rebits]

11/04/202

Design 1

Deployment

If we want to allow local and Jenkins execution of the system tests we must separate deployment/provisioning from the tests themselves. I propose to create a YAML file that specifies the number of hosts, as well as their configuration of a testing system environment.

-
	name: master
	groups: master, manager
	os: ubuntu, centos
	number: 1
	os_default: ['centos', 'centos']
	cpu: 
	vm: 
-
	name: worker
	groups: worker, manager
	os: ubuntu, centos
	number: 2
	default: ['centos', 'centos']
-
	name: agent
	groups: agent
	os: ubuntu, centos, windows, solaris, macos
	number: 3
	default: ['centos', 'centos', 'centos']

Likewise, this structure could be extended to incorporate E2E tests into its logic

-
	name: master-all-in-one
	groups: master, manager, kibana, filebeat, wazuh-dashboard
	os: ubuntu, centos
	number: 1
	os_default: ['centos', 'centos']
	cpu: 2
	vm: 4048
	vars:
		- 
			all_in_one: True

-
	name: agent
	groups: agent
	os: ubuntu, centos, windows, solaris, macos
	number: 1
	default: ['centos']

This file will be located in the corresponding suite of each environment and will be used by QACTL/Jenkins for the deployment of these instances.

With this structure the modules will change from being separated by functionalities without hierarchical order to the following:

system/
├── agentless_cluster
│ └─── deployment.yaml
├─── basic_cluster
│ └└─── deployment.yaml
└── manager_agent.
    └─── deployment.yaml

Interaction with this environment will be through ansible inventory. Jenkins/QACTL will be in charge of generating this inventory once the environment is deployed.

Using this new deployment it will be necessary to:

• Create a library in Jenkins capable of sending instances using this file as input.
• Modify qa_ctl to be able to generate the necessary environment using only this input (modify deployment step).
• Create the necessary tools in Jenkins and qactl to generate the inventory.

Provision

The following basic provisioning playbooks will be created, which will allow eliminating most of the custom playbooks for each of the modules.

• Install Wazuh
• Install all-in-one Wazuh
• Install the wazuh-qa framework
• Configure cluster
• Register agents

These will be located in the system test tools.

The necessary configurations to be made for the module will be made in the test itself.

There will be two different kinds of execution in the local environment:

• Environment already deployed and provisioned:

python3 -m pytest test_system_cluster -i InventoryPath.json

If an inventory is passed, it will be considered that the specified environment has the necessary conditions to launch the test.

Those provisioning tasks specific to each module will be performed by pytest and should leave the environment clean after execution.

• Environment not deployed. In this case, qa_ctl will be in charge of provision and deployment. Package information is required.

python3 -m pytest test_system_cluster -p PackageInformation.json

Provisioning step only will include first of:

• Agent/manager installation on the corresponding hosts.
• Cluster configuration if applicable
• Registration of agents in master/cluster
• Installation of other components if required
• Installation of QA framework

Testing

The Hostmonitor tool will be maintained, improving the tool if it is necessary.
Inventory will be taken directly as a parameter of pytest.

Diagram in Jenkins environment

Cons mitigated using this design:

Deployment

• It has system limitation, only debian can be used
• Deployment and provisioning are performed at the same time
• For every deploy you must navigate to its path
• If we have a wrong environment, we have to destroy it and deploy it again
• Differents environments types cannot be deployed at the same time

Provisioning

• It compiles the source code instead of use a package
• Duplicated snippets -> playbooks, inventory, etc.
• Once deployed and provisioned, you cannot run with another wazuh(-qa) branch

[roronoasins]

Proposal 2

"System tool" that uses inventory data and parsed documentation from the test.

• Module that deploy instances and provision them
• Module that runs the test(s)
• Module that collects logs(and create reports)?(threading) while the test is running, this thread is gathering the logs

Deployment/Provisioning

Using a documentation schema and qa-docs we can obtain some deploy/provision information:

provisioning_type: cluster1

os_platform:
    - linux
    - windows

os_version:
    - Arch Linux
    - Windows Server 2019

• With this info, we can deploy and provision the needed env, getting the initial status.
• Then, every test will have its setup and teardown so the test will run always under the same conditions.
• Also, a health/sanity check would be great so the test does not wait for a callback or any event that wont happend because a previous test broke the env.
• The tool can deploy the machines and destroy after the executions or let them alive just in case we want to keep the env up or debugging or any manual task we want to perform on the remote host.
• Different types of installation? all-in-one, single node, cluster, etc.
• Gather provisioning playbooks or is it possible to autogenerate playbooks using qa-ctl?

Testing framework

• New test structure(create an standard), add/r(efac fixturesthat perform only the expected based in his name. e.g, set_api_default_conf -> only sets the DEFAULT api conf not custom, or restarts wazuh-manager and clean logs)

• With the rework, multiple tests can be run with no problems -> setup/teardown.

Flow

deploy -> provision -> run test -> collect logs(logging, artifacts?) -> cleaning(teardown)

[Rebits]

After a meeting with @wazuh/qa, we have decided to follow #2440 (comment) as a sketch. This will be used as a global vision of the main objectives of the design and it will be changed according to the current developed tools.

The creation of a global tool should be taking into account in the distant future