Integration tests do not run on macOS Arm64

[Deblintrake09]

Description

During the execution of IT test for Release 4.5.1 - Alpha 1 when trying to setup the wazuh-qa repository to execute the ITs on a macOS Monterey Arm64 system, the following issues that caused tests to not be executed, were found :

• It would fail to install cryptography library.

 build/temp.macosx-12-arm64-cpython-310/_openssl.c:575:10: fatal error: 'openssl/opensslv.h' file not found
      #include <openssl/opensslv.h>
               ^~~~~~~~~~~~~~~~~~~~
      1 error generated.
      
          =============================DEBUG ASSISTANCE=============================
          If you are seeing a compilation error please try the following steps to
          successfully install cryptography:
          1) Upgrade to the latest pip and try again. This will fix errors for most
             users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
          2) Read https://cryptography.io/en/latest/installation.html for specific
             instructions for your platform.
          3) Check our frequently asked questions for more information:
             https://cryptography.io/en/latest/faq.html
          =============================DEBUG ASSISTANCE=============================
      
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cryptography
Failed to build cryptography

• tried to fix by updating pip, wheel and setuptools as possible dependencies.

• Installing OpenSSL that is a dependency for cryptography.

• dependencies filetype, lockfile, numpydoc, py, html-reporter, pyyaml would not be installed, but where required for the tests. Installed manually.

• When trying to launch the tests, they would fail due to the html-reporter dependency, that needed to be installed as well. After installing, it tests would still fail to run and not be executed.

sh-3.2# python3.10 -m pytest test_syscollector/
======================================= test session starts ========================================
platform darwin -- Python 3.10.12, pytest-7.4.0, pluggy-1.2.0
rootdir: /Users/vagrant/wazuh-qa/tests/integration
configfile: pytest.ini
plugins: testinfra-8.1.0
collected 27 items                                                                                 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/opt/homebrew/lib/python3.10/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/opt/homebrew/lib/python3.10/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/opt/homebrew/lib/python3.10/site-packages/pluggy/_hooks.py", line 433, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "/opt/homebrew/lib/python3.10/site-packages/pluggy/_manager.py", line 112, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/opt/homebrew/lib/python3.10/site-packages/pluggy/_callers.py", line 155, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/opt/homebrew/lib/python3.10/site-packages/pluggy/_result.py", line 108, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/opt/homebrew/lib/python3.10/site-packages/pluggy/_callers.py", line 80, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/opt/homebrew/lib/python3.10/site-packages/_pytest/main.py", line 334, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/opt/homebrew/lib/python3.10/site-packages/_pytest/main.py", line 671, in perform_collect
INTERNALERROR>     self.config.pluginmanager.check_pending()
INTERNALERROR>   File "/opt/homebrew/lib/python3.10/site-packages/pluggy/_manager.py", line 339, in check_pending
INTERNALERROR>     raise PluginValidationError(
INTERNALERROR> pluggy._manager.PluginValidationError: unknown hook 'pytest_html_results_summary' in plugin <module 'conftest' from '/Users/vagrant/wazuh-qa/tests/integration/conftest.py'>

====================================== no tests ran in 0.05s =======================================
``

### Current behavior
After installing `wazuh-qa` repo requirements and framework to launch ITs, parts of the requirements were not installed or failed to be installed. Tests could not be launched.

### Expected behavior
The tests dependencies and framework are installed correctly and tests can be launched.

[JorgeMarinoDev]

Requested macOS arm64 VMs in internal-devel-requests
https://github.com/wazuh/internal-devel-requests/issues/139
Got access and starting compilation of agent in macOs.
Plan is to reproduce dependency issue running test_syscollector.

[JorgeMarinoDev]

Stats: Able to connect macOs Agent to Manager on VM environment.
Now following rules to setup wazuh-qa integration test to reproduce the issue.
https://github.com/wazuh/wazuh-qa/tree/master/tests/integration#readme

[JorgeMarinoDev]

Status:
While setting up the wazuh-qa in macOS arm64 found the same dependencies issues prior to the pytest_html reported here.
Logging the entire process to report the procedure. Goal: Execute the test_syscollector module successfully.
Possibly the file https://github.com/wazuh/wazuh-qa/tree/v4.5.1-alpha1/tests/integration#readme is outdated and need some changes. Since some deps override the target versions, and others don´t, causing a mismatch. This file is 2 years old.
Logging the entire process to obtain the final deps list for pip3.

[JorgeMarinoDev]

Status:
After following instructions to run integration tests and setting up the environment, several issues were found a priori in a completely isolated venv environment.
After installing wazuh-qa deps package with pip3, tests fail finding the testinfra module.

We keep working on this to get the final pip3 freeze list and procedure from scratch.

[JorgeMarinoDev]

We did get to a resolutive conclusion for this reported issue.

In order to execute integration tests we need to fulfill these instructions:

1. https://github.com/wazuh/wazuh-qa/tree/master/tests/integration#setting-up-a-test-environment (4 moths old)
2. https://github.com/wazuh/wazuh-qa/tree/master/tests/integration#readme (2 years old)

These instructions require a manual procedure to deal with dependencies before executing the individual tests.

pip3 install pytest freezegun jq jsonschema pyyaml==5.4 psutil paramiko distro pandas==0.25.3 pytest-html==2.0.1 numpydoc==0.9.2

As we can see, some modules have a hard coded version number (probably up to date 2 years ago)

The main goal was to be able to execute the test_syscollector suite in macOS arm64 and fix the pip dependencies to do so.

Effectively, an isolated venv was created to do so in a fresh mac agent running and connected to a manager.

Apparently, the integration tests instructions included modules dependencies with now outdated versions. Further on, trying to install them out of the box as the doc states, pip3 reports issues with these versions. That's why this issue itself contains comments that deal with manual installation of some dependencies.

On the other hand, it's pretty common to provide a dependency list with the pip3 freeze command. It lists the currently installed versions of the modules. But over time, with later OS and modules updates, it gets deprecated. Specially, when the list include free and hard coded versions.

In order to get the test_syscollector running , the following procedure was executed with the dependencies listed below.

• macOS agent up and running connected to an Amazon Linux 2 AMI manager.

• Performed changes requested here https://github.com/wazuh/wazuh-qa/tree/master/tests/integration#macos

• Downloaded v4.5.1-alpha1 tag of wazuh-qa repo in agent's host.

• Performed instructions to setup environment https://github.com/wazuh/wazuh-qa/tree/master/tests/integration#setting-up-a-test-environment
  Basically installing pip3 modules from wazuh-qa deps directory. wazuh-testing pip install.
  pip3 install . run in

  /deps/wazuh_testing
  

• Installed pip3 dependencies removing the hardcoded versions stated in https://github.com/wazuh/wazuh-qa/tree/master/tests/integration#readme

  Proposed command:

  pip3 install filetype freezegun jq jsonschema lockfile numpydoc pip-chill psutil pytest-html pytest-testinfra pyyaml
  

• pytest-testinfra is used instead of testinfra (deprecated)

Results

We were able to run the following command without dependency issues:

python3 -m pytest ./test_syscollector

The test session starts running, but displays errors related to the tests themselves. Not related to dependencies.

Next steps

• Modify instructions and remove the hard coded versions of the modules as dependencies.
• Examine causes of failure of test_syscollector. After running, there are several failures related to config parsing. Outside the scope of this issue.

Working Dependencies

Conclusions

• Dependencies issues can be fixed following instructions detailed above,
• test_syscollector failures must be dealt in another issue, probably taking into account output attached here

test_syscollector.txt

[vikman90]

@JorgeMarinoDev thank you!

We would need a PR in this repository to get ITs working!

Best.

[JorgeMarinoDev]

@vikman90 Sounds good. Working on this.
Thank you.

[JorgeMarinoDev]

@vikman90 It'd be really good to boost this https://github.com/wazuh/internal-devel-requests/issues/150

[teddytpc1]

Hi, @JorgeMarinoDev.
Are you still using the requested VM?
Please, let us know if we can terminate the instance.

[JorgeMarinoDev]

@vikman90 @nmkoremblum Tests of PR changes were also performed in MacOs x86. Same procedure. Works OK.

Platform	Result
MacOs ARM	🟢
MacOs x86	🟢