Add test to check new possible flaws in wodles, framework and API code #1659

mcarmona99 · 2021-07-27T12:00:56Z

Related issue
#1615

Hi team,

This PR closes #1615.

In this pull request, I have added a new unit test to check the possible code flaws in our Python.

The unittest uses Bandit to look for new possible vulnerabilities in the wodles, framework and api code.

In order to find new vulnerabilities, the test compares the Bandit output with vulnerabilities we already know.

The test is located at wazuh-qa/tests/security/test_python_code.

Inside this folder, we can find the test itself, called test_python_flaws.py, a raw file called wazuh_branch and a folder called known_flaws.

known_flaws: contains three files. Each line of each file contains a false positive vulnerability detected by Bandit in JSON format. It can also be a known vulnerability that needs to be fixed. These files must be edited after analyzing new vulnerabilities with the test.
wazuh_branch: raw file containing the branch name where we are going to look for vulnerabilities.
test_python_flaws.py: the test itself. This test is going to be passed using the same environment used in the Wazuh framework and API unittests. If the test fails, new txt files will be created in wazuh-qa/tests/security/test_python_code showing information about the new vulnerabilities found.

So, the workaround for this test will be: passing the test, if it passes, everything is correct. If the test fails, new code vulnerabilities will be found in wazuh-qa/tests/security/test_python_code/new_flaws_{module}.txt. After analyzing this vulnerabilities there are 2 options:

We move them to its respective known_flaws.txt file if we consider it is a false positive. Note that the JSON showing the new flaw is prettified and has line number, we should replace the line number in the code key to * and unprettify the JSON.
We will change the new code to avoid the vulnerability.

Test output example:

test_output

ytest tests/security/test_python_code/test_python_flaws.py 
============================================================= test session starts =============================================================
platform linux -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/manuel/git/wazuh-qa
plugins: cov-2.12.0, asyncio-0.14.0
collected 3 items                                                                                                                             

tests/security/test_python_code/test_python_flaws.py ..F                                                                                [100%]

================================================================== FAILURES ===================================================================
_______________________________________________________ test_check_security_flaws[api] ________________________________________________________

clone_wazuh_repository = '/tmp/tmpravk09cy', directory_to_check = 'api'

    @pytest.mark.parametrize("directory_to_check", DIRECTORIES_TO_CHECK)
    def test_check_security_flaws(clone_wazuh_repository, directory_to_check):
        # Wazuh is cloned from GitHub using the clone_wazuh_repository fixture
        assert clone_wazuh_repository, f"Error while cloning the Wazuh repository from GitHub, " \
                                       f"please check the wazuh branch in the {WAZUH_BRANCH_FILE} file"
    
        # run Bandit to check possible security flaws
        bandit_output = json.loads(
            os.popen(f"bandit -q -r {clone_wazuh_repository}/{directory_to_check} "
                     f"-ii -f {FORMAT} -x {','.join(DIRECTORIES_TO_EXCLUDE)}").read())
    
        assert not bandit_output['errors'], \
            f"\nBandit returned errors when trying to get possible vulnerabilities:\n{bandit_output['errors']}"
    
        # We save the results obtained in the report as the rest of information is redundant or not used
        original_results = bandit_output['results']
    
        # Delete filenames to make it persistent with tmp directories
        for result in original_results:
            result['filename'] = "/".join(result['filename'].split('/')[3:])
    
        results = copy.deepcopy(original_results)
    
        # Delete line numbers in code to make it persistent with updates
        for result in results:
            code = result['code'].split("\n")[:-1]
            for i in range(len(code)):
                code[i] = re.sub(r"^\d+", "*", code[i])
            # Join the modified code as it was done before splitting
            result['code'] = '\n'.join(code)
    
        # Compare the flaws obtained in results with the known flaws
        with open(f"{KNOWN_FLAWS_DIRECTORY}/known_flaws_{directory_to_check}.txt", mode="r") as f:
            file_content = f.read().split("\n")
            known_flaws = []
            for line in file_content:
                try:
                    known_flaws.append(json.loads(line))
                except json.JSONDecodeError:
                    pass
    
        # There are security flaws if there are new possible vulnerabilities detected
        new_flaws = [flaw for flaw in results if flaw not in known_flaws]
        if new_flaws:
            # Change new_flaws to the original flaws reported (with line numbers)
            for i in range(len(new_flaws)):
                new_flaws[i] = original_results[results.index(new_flaws[i])]
    
            # Write new flaws in a temporal file to analyze them
            new_flaws_path = os.path.join(TEST_PYTHON_CODE_PATH, f"new_flaws_{directory_to_check}.txt")
            with open(new_flaws_path, mode="w") as f:
                for flaw in new_flaws:
                    f.write(json.dumps(flaw, indent=4, sort_keys=True))
                    f.write("\n")
            files_with_flaws = ', '.join(list(dict.fromkeys([res['filename'] for res in new_flaws])))
>           assert False, f"\nVulnerabilities found in files:\n{files_with_flaws}" \
                          f"\nCheck them in {new_flaws_path}"
E           AssertionError: 
E             Vulnerabilities found in files:
E             api/scripts/wazuh-apid.py
E             Check them in /home/manuel/git/wazuh-qa/tests/security/test_python_code/new_flaws_api.txt
E           assert False

tests/security/test_python_code/test_python_flaws.py:103: AssertionError
=========================================================== short test summary info ===========================================================
FAILED tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws[api] - AssertionError: 
======================================================== 1 failed, 2 passed in 50.52s =========================================================

And the vulnerability detected is:

{
    "code": "153                                )\n154     app.add_api('spec.yaml',\n155                 arguments={'title': 'Wazuh API',\n156                            'protocol': 'https' if api_conf['https']['enabled'] else 'http',\n157                            'host': api_conf['host'],\n158                            'port': api_conf['port']\n159                            },\n160                 strict_validation=True,\n161                 validate_responses=False,\n162                 pass_context_arg_name='request',\n163                 options={\"middlewares\": [response_postprocessing, set_user_name, security_middleware, request_logging,\n164                                          set_secure_headers]})\n165 \n",
    "filename": "api/scripts/wazuh-apid.py",
    "issue_confidence": "MEDIUM",
    "issue_severity": "LOW",
    "issue_text": "Possible hardcoded password: 'request'",
    "line_number": 154,
    "line_range": [
        154,
        155,
        156,
        157,
        158,
        159,
        160,
        161,
        162,
        163,
        164
    ],
    "more_info": "https://bandit.readthedocs.io/en/latest/plugins/b106_hardcoded_password_funcarg.html",
    "test_id": "B106",
    "test_name": "hardcoded_password_funcarg"
}

After analyzing the possible flaw, we see it is a false positive, we move it to known_flaws/known_flaws_api.txt.

known_flaws/known_flaws_api.txt

False positives:
{"code": "*                                )\n*     app.add_api('spec.yaml',\n*                 arguments={'title': 'Wazuh API',\n*                            'protocol': 'https' if api_conf['https']['enabled'] else 'http',\n*                            'host': api_conf['host'],\n*                            'port': api_conf['port']\n*                            },\n*                 strict_validation=True,\n*                 validate_responses=False,\n*                 pass_context_arg_name='request',\n*                 options={\"middlewares\": [response_postprocessing, set_user_name, security_middleware, request_logging,\n*                                          set_secure_headers]})\n* ", "filename": "api/scripts/wazuh-apid.py", "issue_confidence": "MEDIUM", "issue_severity": "LOW", "issue_text": "Possible hardcoded password: 'request'", "line_number": 154, "line_range": [154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164], "more_info": "https://bandit.readthedocs.io/en/latest/plugins/b106_hardcoded_password_funcarg.html", "test_id": "B106", "test_name": "hardcoded_password_funcarg"}

If we pass the test again, it will pass.

$ pytest tests/security/test_python_code/test_python_flaws.py 
============================================================= test session starts =============================================================
platform linux -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/manuel/git/wazuh-qa
plugins: cov-2.12.0, asyncio-0.14.0
collected 3 items                                                                                                                             

tests/security/test_python_code/test_python_flaws.py ...                                                                                [100%]

============================================================= 3 passed in 42.48s ==============================================================

Regards,
Manuel.

Tests

Proven that tests pass when they have to pass.
Proven that tests fail when they have to fail.

Python codebase satisfies PEP-8 style style guide. pycodestyle --max-line-length=120 --show-source --show-pep8 file.py.
Python codebase is documented following the Google Style for Python docstrings.
The test is documented in wazuh-qa/docs.
provision_documentation.sh generate the docs without errors.

… code

…changed to JSON

mcarmona99 · 2021-07-29T15:02:51Z

PR UPDATE

The branch name is now passed as a parameter to the test python script:

pytest tests/security/test_python_code/test_python_flaws.py --branch <branch>

eg: pytest tests/security/test_python_code/test_python_flaws.py --branch master

Because of this change, the wazuh_branch file has been removed.

Line numbers have been removed from code and if line_number and line_range change, this changes will be done in the known_flaws files to maintain it updated. If a known flaw was fixed, the test will automatically remove it from new_flaws.
The output of the test is now JSON files. The files to compare the new files are also JSON files. This will be useful in order to automate this test in the future.

Format of the new flaws generated by the test:

{
    "new_flaws": [
    { # flaw 1
    ...
    },
    { # flaw 2
    ...
    },
    
    ...
    
    { # flaw n
    ...
    },
}

Example:

{
    "new_flaws": [
        {
            "code": "24 from subprocess import CalledProcessError, check_output\n25 from xml.etree.ElementTree import ElementTree\n26 \n27 from cachetools import cached, TTLCache\n",
            "filename": "framework/wazuh/core/utils.py",
            "issue_confidence": "HIGH",
            "issue_severity": "LOW",
            "issue_text": "Using ElementTree to parse untrusted XML data is known to be vulnerable to XML attacks. Replace ElementTree with the equivalent defusedxml package, or make sure defusedxml.defuse_stdlib() is called.",
            "line_number": 25,
            "line_range": [
                25,
                26
            ],
            "more_info": "https://bandit.readthedocs.io/en/latest/blacklists/blacklist_imports.html#b405-import-xml-etree",
            "test_id": "B405",
            "test_name": "blacklist"
        },
        {
            "code": "111     try:\n112         output = check_output(command)\n113     except CalledProcessError as error:\n",
            "filename": "framework/wazuh/core/utils.py",
            "issue_confidence": "HIGH",
            "issue_severity": "LOW",
            "issue_text": "subprocess call - check for execution of untrusted input.",
            "line_number": 112,
            "line_range": [
                112
            ],
            "more_info": "https://bandit.readthedocs.io/en/latest/plugins/b603_subprocess_without_shell_equals_true.html",
            "test_id": "B603",
            "test_name": "subprocess_without_shell_equals_true"
        },
        {
            "code": "640 def md5(fname):\n641     hash_md5 = hashlib.md5()\n642     with open(fname, \"rb\") as f:\n",
            "filename": "framework/wazuh/core/utils.py",
            "issue_confidence": "HIGH",
            "issue_severity": "MEDIUM",
            "issue_text": "Use of insecure MD2, MD4, MD5, or SHA1 hash function.",
            "line_number": 641,
            "line_range": [
                641
            ],
            "more_info": "https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b303-md5",
            "test_id": "B303",
            "test_name": "blacklist"
        }
    ]
}

I have removed the subprocess calls. Now the Bandit run is done by calling the python package itself. Same for the git calls.
Updated docstring to follow the Google standard.

Test result:

pytest tests/security/test_python_code/test_python_flaws.py  --branch master
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/manuel/git/wazuh-qa
plugins: cov-2.12.0, asyncio-0.14.0
collected 3 items                                                              

tests/security/test_python_code/test_python_flaws.py ...                 [100%]

============================== 3 passed in 37.57s ==============================

Regards,
Manuel.

mcarmona99 · 2021-07-30T15:06:02Z

PR update

I have included parameters to add the possibility to use this test with more repositories, directories, minimum confidence and severity level and different excluded directories.

These are the parameters:

	--branch -> set branch of the repository
	--repo -> set repository inside the Wazuh organization
	--check_directories -> set the directories to check, this must be a string with the directory name. If more than one is indicated, they must be separated with comma.
	--exclude_directories -> set the directories to exclude, this must be a string with the directory name. If more than one is indicated, they must be separated with comma.
	--confidence -> set the minimum value of confidence of the Bandit scan. This value must be 'UNDEFINED', 'LOW', 'MEDIUM' or 'HIGH'
	--severity -> set the minimum value of severity of the Bandit scan. This value must be 'UNDEFINED', 'LOW', 'MEDIUM' or 'HIGH'

Default parameters:

branch='master'
repository='wazuh'
check_directories='framework/,wodles/,api/'
excluded_directories='tests/,test/'
confidence='MEDIUM'
severity='LOW'

mcarmona99 · 2021-08-02T13:18:46Z

PR update

The test works with all the default values of the parameters.

I have included a fix as the exclude_directories parameter was not working properly when running the test from a different directory the test is.

Parameters we have:

--repo 
--branch

The test will fail if the repository or branch are not valid.

--check_directories
--exclude_directories
--confidence
--severity

--exclude_directories: had an error, this parameters was not working when launching the test in a directory out of the one containing the test. I haved fixed this and now it works.
--branch: working, the test will pass when the code added in that branch does not have possible flaws.
--repo, --check_directories, --exclude_directories, --confidence, --severity: working. The test will always fail if we use at least one of these parameters without the default values. The new flaws will appear in the same level the script is:

Example of the test with the wazuh-qa repository and a directory of this repo.

example 1

pytest tests/security/test_python_code/test_python_flaws.py --repo wazuh-qa --check_directories deps/
========================================================== test session starts ==========================================================
platform linux -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/manuel/git/wazuh-qa
plugins: cov-2.12.0, asyncio-0.14.0
collected 1 item                                                                                                                        

tests/security/test_python_code/test_python_flaws.py F                                                                            [100%]

=============================================================== FAILURES ================================================================
_______________________________________________________ test_check_security_flaws _______________________________________________________

clone_wazuh_repository = '/tmp/tmprwu64v74'
test_parameters = {'directories_to_check': ['deps/'], 'directories_to_exclude': 'tests/,test/', 'min_confidence_level': 'MEDIUM', 'min_severity_level': 'LOW', ...}

    def test_check_security_flaws(clone_wazuh_repository, test_parameters):
        """Test whether the directory to check has python files with possible vulnerabilities or not.
    
        The test passes if there are no new vulnerabilities. The test fails in other case and generates a report.
    
        In case there is at least one vulnerability, a json file will be generated with the report. If we consider this
        result or results are false positives, we will move the json object containing each specific result to the
        `known_flaws/known_flaws_{framework|api|wodles}.json` file.
    
        Args:
            clone_wazuh_repository (fixture): Pytest fixture returning the path of the temporary directory path the
                repository cloned. This directory is removed at the end of the pytest session.
            test_parameters (fixture): Pytest fixture returning the a dictionary with all the test parameters.
                These parameters are the directories to check, directories to exclude, the minimum confidence level, the
                minimum severity level and the repository name.
        """
        # Wazuh is cloned from GitHub using the clone_wazuh_repository fixture
        assert clone_wazuh_repository, "Error while cloning the Wazuh repository from GitHub, " \
                                       "please check the Wazuh branch set in the parameter."
        # Change to the cloned Wazuh repository directory
        os.chdir(clone_wazuh_repository)
    
        flaws_found = {directory: None for directory in test_parameters['directories_to_check']}
        for directory_to_check in test_parameters['directories_to_check']:
            is_default_check_dir = directory_to_check.replace('/', '') in \
                                   DEFAULT_DIRECTORIES_TO_CHECK.replace('/', '').split(',') and test_parameters[
                                       'repository'] == DEFAULT_REPOSITORY
            # Run Bandit scan
            bandit_output = run_bandit_scan(directory_to_check,
                                            test_parameters['directories_to_exclude'],
                                            test_parameters['min_severity_level'],
                                            test_parameters['min_confidence_level'])
            assert not bandit_output['errors'], \
                f"\nBandit returned errors when trying to get possible vulnerabilities in the directory " \
                f"{directory_to_check}:\n{bandit_output['errors']}"
    
            # We save the results obtained in the report as the rest of information is redundant or not used
            results = bandit_output['results']
    
            # Delete line numbers in code to make it persistent with updates
            for result in results:
                result['code'] = re.sub(r"^\d+", "", result['code'])  # Delete first line number
                result['code'] = re.sub(r"\n\d+", "\n", result['code'], re.M)  # Delete line numbers after newline
    
            # Compare the flaws obtained in results with the known flaws
            if is_default_check_dir:
                try:
                    with open(f"{KNOWN_FLAWS_DIRECTORY}/known_flaws_{directory_to_check.replace('/', '')}.json",
                              mode="r") as f:
                        known_flaws = json.load(f)
                except json.decoder.JSONDecodeError or FileNotFoundError:
                    known_flaws = {'false_positives': [], 'to_fix': []}
            else:
                known_flaws = {'false_positives': [], 'to_fix': []}
    
            # There are security flaws if there are new possible vulnerabilities detected
            # To compare them, we cannot compare the whole dictionaries containing the flaws as the values of keys like
            # line_number and line_range will vary
            # Update known flaws with the ones detected in this Bandit run, remove them if they were fixed
            known_flaws = update_known_flaws(known_flaws, results)
            if is_default_check_dir:
                with open(f"{KNOWN_FLAWS_DIRECTORY}/known_flaws_{directory_to_check.replace('/', '')}.json", mode="w") as f:
                    f.write(json.dumps(known_flaws, indent=4, sort_keys=True))
            else:
                # if the directory to check is not one of the default list, we will create a new known_flaws file outside
                # the directory known_flaws, to avoid overwriting
                with open(f"known_flaws_{directory_to_check.replace('/', '')}.json", mode="w") as f:
                    f.write(json.dumps(known_flaws, indent=4, sort_keys=True))
    
            new_flaws = [flaw for flaw in results if
                         flaw not in known_flaws['to_fix'] and flaw not in known_flaws['false_positives']]
            if new_flaws:
                # Write new flaws in a temporal file to analyze them
                new_flaws_path = os.path.join(TEST_PYTHON_CODE_PATH,
                                              f"new_flaws_{directory_to_check.replace('/', '')}.json")
                with open(new_flaws_path, mode="w+") as f:
                    f.write(json.dumps({'new_flaws': new_flaws}, indent=4, sort_keys=True))
                files_with_flaws = ', '.join(list(dict.fromkeys([res['filename'] for res in new_flaws])))
                flaws_found[directory_to_check] = f"Vulnerabilities found in files: {files_with_flaws}," \
                                                  f" check them in {new_flaws_path}"
    
>       assert not any(flaws_found[directory] for directory in test_parameters['directories_to_check']), \
            f"\nThe following possible vulnerabilities were found: {json.dumps(flaws_found, indent=4, sort_keys=True)}"
E       AssertionError: 
E         The following possible vulnerabilities were found: {
E             "deps/": "Vulnerabilities found in files: deps/wazuh_testing/wazuh_testing/agent.py, deps/wazuh_testing/wazuh_testing/analysis.py, deps/wazuh_testing/wazuh_testing/api.py, deps/wazuh_testing/wazuh_testing/fim.py, deps/wazuh_testing/wazuh_testing/logcollector.py, deps/wazuh_testing/wazuh_testing/remote.py, deps/wazuh_testing/wazuh_testing/scripts/simulate_api_load.py, deps/wazuh_testing/wazuh_testing/tools/__init__.py, deps/wazuh_testing/wazuh_testing/tools/agent_simulator.py, deps/wazuh_testing/wazuh_testing/tools/api_simulator.py, deps/wazuh_testing/wazuh_testing/tools/configuration.py, deps/wazuh_testing/wazuh_testing/tools/file.py, deps/wazuh_testing/wazuh_testing/tools/key_polling_api.py, deps/wazuh_testing/wazuh_testing/tools/key_polling_cluster.py, deps/wazuh_testing/wazuh_testing/tools/remoted_sim.py, deps/wazuh_testing/wazuh_testing/tools/security.py, deps/wazuh_testing/wazuh_testing/tools/services.py, deps/wazuh_testing/wazuh_testing/tools/system.py, deps/wazuh_testing/wazuh_testing/tools/time.py, deps/wazuh_testing/wazuh_testing/tools/utils.py, deps/wazuh_testing/wazuh_testing/vulnerability_detector.py, check them in /home/manuel/git/wazuh-qa/tests/security/test_python_code/new_flaws_deps.json"
E         }
E       assert not True
E        +  where True = any(<generator object test_check_security_flaws.<locals>.<genexpr> at 0x7f040f0200b0>)

/home/manuel/git/wazuh-qa/tests/security/test_python_code/test_python_flaws.py:220: AssertionError
=========================================================== warnings summary ============================================================
tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
  invalid escape sequence \[

tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
  invalid escape sequence \]

tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
  invalid escape sequence \<

tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
  invalid escape sequence \>

tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
  invalid escape sequence \s

-- Docs: https://docs.pytest.org/en/stable/warnings.html
======================================================== short test summary info ========================================================
FAILED tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws - AssertionError: 
==================================================== 1 failed, 10 warnings in 42.72s ====================================================

Example of the test with the wazuh-qa repository and a directory of this repo with no security flaws (the test passes).

example 2

pytest tests/security/test_python_code/test_python_flaws.py --repo wazuh-qa --check_directories deps/ --exclude_directories wazuh_testing/
========================================================== test session starts ==========================================================
platform linux -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/manuel/git/wazuh-qa
plugins: cov-2.12.0, asyncio-0.14.0
collected 1 item                                                                                                                        

tests/security/test_python_code/test_python_flaws.py .                                                                            [100%]

===================================================== 1 passed in 178.32s (0:02:58) =====================================================

Example of the test with the wazuh repo and the default values but with a different branch (vulnerability in wodle and known_flaws updated (new lines)).

example 3

pytest tests/security/test_python_code/test_python_flaws.py --repo wazuh --check_directories wodles/,framework/,api/ --exclude_directories tests/,test/ --branch qa-1615-to-delete
========================================================== test session starts ==========================================================
platform linux -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/manuel/git/wazuh-qa
plugins: cov-2.12.0, asyncio-0.14.0
collected 1 item                                                                                                                        

tests/security/test_python_code/test_python_flaws.py F                                                                            [100%]

=============================================================== FAILURES ================================================================
_______________________________________________________ test_check_security_flaws _______________________________________________________

clone_wazuh_repository = '/tmp/tmp3_miyr4f'
test_parameters = {'directories_to_check': ['wodles/', 'framework/', 'api/'], 'directories_to_exclude': 'tests/,test/', 'min_confidence_level': 'MEDIUM', 'min_severity_level': 'LOW', ...}

    def test_check_security_flaws(clone_wazuh_repository, test_parameters):
        """Test whether the directory to check has python files with possible vulnerabilities or not.
    
        The test passes if there are no new vulnerabilities. The test fails in other case and generates a report.
    
        In case there is at least one vulnerability, a json file will be generated with the report. If we consider this
        result or results are false positives, we will move the json object containing each specific result to the
        `known_flaws/known_flaws_{framework|api|wodles}.json` file.
    
        Args:
            clone_wazuh_repository (fixture): Pytest fixture returning the path of the temporary directory path the
                repository cloned. This directory is removed at the end of the pytest session.
            test_parameters (fixture): Pytest fixture returning the a dictionary with all the test parameters.
                These parameters are the directories to check, directories to exclude, the minimum confidence level, the
                minimum severity level and the repository name.
        """
        # Wazuh is cloned from GitHub using the clone_wazuh_repository fixture
        assert clone_wazuh_repository, "Error while cloning the Wazuh repository from GitHub, " \
                                       "please check the Wazuh branch set in the parameter."
        # Change to the cloned Wazuh repository directory
        os.chdir(clone_wazuh_repository)
    
        flaws_found = {}
        for directory_to_check in test_parameters['directories_to_check']:
            is_default_check_dir = directory_to_check.replace('/', '') in \
                                   DEFAULT_DIRECTORIES_TO_CHECK.replace('/', '').split(',') and test_parameters[
                                       'repository'] == DEFAULT_REPOSITORY
            # Run Bandit scan
            bandit_output = run_bandit_scan(directory_to_check,
                                            test_parameters['directories_to_exclude'],
                                            test_parameters['min_severity_level'],
                                            test_parameters['min_confidence_level'])
            assert not bandit_output['errors'], \
                f"\nBandit returned errors when trying to get possible vulnerabilities in the directory " \
                f"{directory_to_check}:\n{bandit_output['errors']}"
    
            # We save the results obtained in the report as the rest of information is redundant or not used
            results = bandit_output['results']
    
            # Delete line numbers in code to make it persistent with updates
            for result in results:
                result['code'] = re.sub(r"^\d+", "", result['code'])  # Delete first line number
                result['code'] = re.sub(r"\n\d+", "\n", result['code'], re.M)  # Delete line numbers after newline
    
            # Compare the flaws obtained in results with the known flaws
            if is_default_check_dir:
                try:
                    with open(f"{KNOWN_FLAWS_DIRECTORY}/known_flaws_{directory_to_check.replace('/', '')}.json",
                              mode="r") as f:
                        known_flaws = json.load(f)
                except json.decoder.JSONDecodeError or FileNotFoundError:
                    known_flaws = {'false_positives': [], 'to_fix': []}
            else:
                known_flaws = {'false_positives': [], 'to_fix': []}
    
            # There are security flaws if there are new possible vulnerabilities detected
            # To compare them, we cannot compare the whole dictionaries containing the flaws as the values of keys like
            # line_number and line_range will vary
            # Update known flaws with the ones detected in this Bandit run, remove them if they were fixed
            known_flaws = update_known_flaws(known_flaws, results)
            if is_default_check_dir:
                with open(f"{KNOWN_FLAWS_DIRECTORY}/known_flaws_{directory_to_check.replace('/', '')}.json", mode="w") as f:
                    f.write(json.dumps(known_flaws, indent=4, sort_keys=True))
            else:
                # if the directory to check is not one of the default list, we will create a new known_flaws file outside
                # the directory known_flaws, to avoid overwriting
                with open(f"known_flaws_{directory_to_check.replace('/', '')}.json", mode="w") as f:
                    f.write(json.dumps(known_flaws, indent=4, sort_keys=True))
    
            new_flaws = [flaw for flaw in results if
                         flaw not in known_flaws['to_fix'] and flaw not in known_flaws['false_positives']]
            if new_flaws:
                # Write new flaws in a temporal file to analyze them
                new_flaws_path = os.path.join(TEST_PYTHON_CODE_PATH,
                                              f"new_flaws_{directory_to_check.replace('/', '')}.json")
                with open(new_flaws_path, mode="w+") as f:
                    f.write(json.dumps({'new_flaws': new_flaws}, indent=4, sort_keys=True))
                files_with_flaws = ', '.join(list(dict.fromkeys([res['filename'] for res in new_flaws])))
                flaws_found[directory_to_check] = f"Vulnerabilities found in files: {files_with_flaws}," \
                                                  f" check them in {new_flaws_path}"
    
>       assert not any(flaws_found[directory] for directory in test_parameters['directories_to_check']), \
            f"\nThe following possible vulnerabilities were found: {json.dumps(flaws_found, indent=4, sort_keys=True)}"
E       AssertionError: 
E         The following possible vulnerabilities were found: {
E             "wodles/": "Vulnerabilities found in files: wodles/azure/azure-logs.py, check them in /home/manuel/git/wazuh-qa/tests/security/test_python_code/new_flaws_wodles.json"
E         }
E       assert not True
E        +  where True = any(<generator object test_check_security_flaws.<locals>.<genexpr> at 0x7ffb9c0fef90>)

/home/manuel/git/wazuh-qa/tests/security/test_python_code/test_python_flaws.py:220: AssertionError
=========================================================== warnings summary ============================================================
tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
  invalid escape sequence \S

tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
  invalid escape sequence \s

tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws
  invalid escape sequence \g

-- Docs: https://docs.pytest.org/en/stable/warnings.html
======================================================== short test summary info ========================================================
FAILED tests/security/test_python_code/test_python_flaws.py::test_check_security_flaws - AssertionError: 
=============================================== 1 failed, 3 warnings in 189.58s (0:03:09) ===============================================

The test will pass with the default parameters. It will also pass with a different branch if there are no new possible flaws. When changing the rest of parameters, the test will fail if new flaws are detected.

README.md

docs/index.md

docs/tests/index.md

tests/security/test_python_code/conftest.py

tests/security/conftest.py

tests/security/test_python_code/known_flaws/known_flaws_framework.json

Rebits · 2021-08-26T09:27:57Z

This test seems to work correctly, running without any fail if python3.9.6 is used and it fails for python3.6

Local-3.9.6	Status	Report
R1	🟢	R1
R2	🟢	R2
R3	🟢	R3

Local-3.6.8	Status	Report
R	🔴	R

In order to use this test in the Jenkins environment, it should be necessary to install a proper python version, otherwise, the test will fail. Also, it could affect the development tool qactl. We should consider the python version in environment provision.

Rebits

GJ, but some changes are required

tests/security/test_python_code/test_python_flaws.py

Rebits · 2021-08-26T11:17:42Z

Final results after review comments applied.

Local-3.9.6	Status	Report
R1	🟢	R1
R2	🟢	R2
R3	🟢	R3

Rebits

LGTM

AdriiiPRodri

LGTM

Add unittest to check new possible flaws in wodles, framework and api…

5766318

… code

mcarmona99 self-assigned this Jul 27, 2021

mcarmona99 added 7 commits July 28, 2021 13:33

Add parameter to set branch in the flaw test, output and known_flaws …

beb0f7c

…changed to JSON

Move conftest to a external level

a2abb64

Minor change to string

61e1ce7

Fix error when updating known_flaws files

0f136a4

Change bandit call to use the bandit dep instead of subprocess

1e38c1d

Update git calls to use the git Python library

edeb388

Update docstrings

6765490

mcarmona99 added 3 commits July 30, 2021 17:17

Add extra parameters to the test

0867571

Minor changes to the test

f776317

Minor change to assert false output

c924b8c

Minor change to fix an error

a2f8d91

mcarmona99 force-pushed the feature/1615-python-code-flaws-test branch from 320aed5 to a2f8d91 Compare August 10, 2021 09:51

mcarmona99 added 5 commits August 10, 2021 12:13

Add docs for the new test_python_code

16362eb

Update test_python_flaws documentation

4095c77

Minor change to mkdocs

913c5cc

Add README to test_python_code directory

43c6495

Add bandit to requirements

69da873

mcarmona99 marked this pull request as ready for review August 11, 2021 09:34

AdriiiPRodri suggested changes Aug 11, 2021

View reviewed changes

mcarmona99 added 2 commits August 12, 2021 10:49

Add review changes to the Python code test

0a0f1dd

Add depth=1 to git clone command

beddbc1

mcarmona99 requested a review from AdriiiPRodri August 12, 2021 09:20

AdriiiPRodri previously approved these changes Aug 12, 2021

View reviewed changes

AdriiiPRodri requested a review from davidjiglesias August 12, 2021 11:10

davidjiglesias changed the title ~~Add unittest to check new possible flaws in wodles, framework and api code~~ Add test to check new possible flaws in wodles, framework and api code Aug 20, 2021

Rebits requested changes Aug 26, 2021

View reviewed changes

Minor changes on variable names and docstring

b9dd7f6

mcarmona99 dismissed AdriiiPRodri’s stale review via b9dd7f6 August 26, 2021 10:50

mcarmona99 requested a review from Rebits August 26, 2021 10:50

Rebits previously approved these changes Aug 26, 2021

View reviewed changes

Update python code test and functions changed to tools

14e9787

mcarmona99 dismissed Rebits’s stale review via 14e9787 September 10, 2021 06:46

mcarmona99 added 2 commits September 10, 2021 08:53

Update docs for the code analysis test

b3bb4fe

Change false_positives to to_fix after creating its respective issues

075faf4

mcarmona99 requested review from Rebits and AdriiiPRodri September 14, 2021 08:22

Kondent mentioned this pull request Sep 14, 2021

Add test to scan all python packages #1652

Merged

mcarmona99 added 2 commits September 16, 2021 10:28

Update docs for the Python vulnerabilities test

abcfbb5

Update known flaws after creating the issues

6ebd520

AdriiiPRodri approved these changes Sep 16, 2021

View reviewed changes

davidjiglesias changed the title ~~Add test to check new possible flaws in wodles, framework and api code~~ Add test to check new possible flaws in wodles, framework and API code Sep 23, 2021

snaow merged commit a4ce1ff into master Oct 1, 2021

snaow deleted the feature/1615-python-code-flaws-test branch October 1, 2021 08:05

snaow mentioned this pull request Jan 27, 2022

QA Release - Rev 430031 #2500

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add test to check new possible flaws in wodles, framework and API code #1659

Add test to check new possible flaws in wodles, framework and API code #1659

mcarmona99 commented Jul 27, 2021 •

edited

Loading

mcarmona99 commented Jul 29, 2021

mcarmona99 commented Jul 30, 2021 •

edited

Loading

mcarmona99 commented Aug 2, 2021 •

edited

Loading

Rebits commented Aug 26, 2021

Rebits left a comment

Rebits commented Aug 26, 2021

Rebits left a comment

AdriiiPRodri left a comment

Add test to check new possible flaws in wodles, framework and API code #1659

Add test to check new possible flaws in wodles, framework and API code #1659

Conversation

mcarmona99 commented Jul 27, 2021 • edited Loading

Tests

mcarmona99 commented Jul 29, 2021

PR UPDATE

mcarmona99 commented Jul 30, 2021 • edited Loading

PR update

mcarmona99 commented Aug 2, 2021 • edited Loading

PR update

Rebits commented Aug 26, 2021

Rebits left a comment

Choose a reason for hiding this comment

Rebits commented Aug 26, 2021

Rebits left a comment

Choose a reason for hiding this comment

AdriiiPRodri left a comment

Choose a reason for hiding this comment

mcarmona99 commented Jul 27, 2021 •

edited

Loading

mcarmona99 commented Jul 30, 2021 •

edited

Loading

mcarmona99 commented Aug 2, 2021 •

edited

Loading