Merge pull request #10487 from rhmdnd/add-requirements-file

Add requirements files for python dependencies
ComplianceAsCode · Jun 6, 2023 · dad8550 · dad8550
2 parents e20576f + e014370
commit dad8550
Show file tree

Hide file tree

Showing 4 changed files with 53 additions and 42 deletions.
diff --git a/.github/workflows/gate.yaml b/.github/workflows/gate.yaml
@@ -58,11 +58,11 @@ jobs:
       - name: Update the package repository
         run: apt-get update
       - name: Install Deps
-        run: apt-get install -y ansible-lint bats check cmake libopenscap8 libxml2-utils ninja-build python3-github python3-jinja2 python3-setuptools python3-pip python3-pytest python3-pytest-cov xsltproc
-      - name: Install deps python
-        run: pip3 install ruamel.yaml yamlpath pyyaml==5.4.1
+        run: apt-get install -y ansible-lint bats check cmake libopenscap8 libxml2-utils ninja-build python3-github python3-pip xsltproc
       - name: Checkout
         uses: actions/checkout@v3
+      - name: Install deps python
+        run: pip3 install -r requirements.txt -r test-requirements.txt
       - name: Build
         env:
           ADDITIONAL_CMAKE_OPTIONS: "-DSSG_ANSIBLE_PLAYBOOKS_PER_RULE_ENABLED=ON"
@@ -129,11 +129,11 @@ jobs:
       image: fedora:latest
     steps:
       - name: Install Deps
-        run: dnf install -y cmake make openscap-utils python3-pyyaml python3-setuptools python3-jinja2 bats python3-pytest python3-pytest-cov ansible python3-pip ShellCheck git python3-mypy python3-lxml
-      - name: Install deps python
-        run: pip install ruamel.yaml yamlpath types-openpyxl types-PyYAML xmldiff openpyxl openpyxl-stubs pandas cmakelint
+        run: dnf install -y cmake make openscap-utils python3-pyyaml bats ansible python3-pip ShellCheck git
       - name: Checkout
         uses: actions/checkout@v3
+      - name: Install deps python
+        run: pip install -r requirements.txt -r test-requirements.txt
       - name: Build
         run: |-
           ./build_product \
@@ -179,11 +179,11 @@ jobs:
       -   name: Run Updates
           run: dnf update -y
       -   name: Install Deps
-          run: dnf install -y cmake make openscap-utils python3-pyyaml python3-jinja2 bats python3-pytest python3-pytest-cov ansible python3-pip ShellCheck python-mypy git
-      -   name: Install deps python
-          run: pip install ruamel.yaml yamlpath types-openpyxl types-PyYAML openpyxl openpyxl-stubs pandas
+          run: dnf install -y cmake make openscap-utils bats ansible python3-pip ShellCheck git
       -   name: Checkout
           uses: actions/checkout@v3
+      -   name: Install deps python
+          run: pip install -r requirements.txt -r test-requirements.txt
       -   name: Build
           run: |-
             ./build_product \

diff --git a/docs/manual/developer/02_building_complianceascode.md b/docs/manual/developer/02_building_complianceascode.md
@@ -30,29 +30,45 @@ less ~/OpenSCAP/STARTGUIDE.md
 ## Installing build dependencies
 
 ### Required Dependencies
-On *Red Hat Enterprise Linux 7* make sure the packages `cmake`, `openscap-utils`,
-`PyYAML`, `python-jinja2`, `python-setuptools` and their dependencies are installed:
+On *Red Hat Enterprise Linux 7* make sure the following packages are installed:
 
 ```bash
-yum install cmake make openscap-utils openscap-scanner PyYAML python-jinja2
+yum install cmake make openscap-utils openscap-scanner
 ```
 
-On *Red Hat Enterprise Linux 8* and *Fedora* the package list is the same but python2 packages need to be replaced with python3 ones:
+On *Red Hat Enterprise Linux 8* and *Fedora* the package list but must also include python3:
 
 ```bash
-yum install cmake make openscap-utils openscap-scanner python3 python3-pyyaml python3-jinja2 python3-setuptools
+yum install cmake make openscap-utils openscap-scanner python3
 ```
 
 On *Ubuntu* and *Debian*, make sure the packages `libopenscap8`,
-`libxml2-utils`, `python3-jinja2`, `python3-yaml`, `python3-setuptools`, `xsltproc` and their dependencies are
-installed:
+`libxml2-utils`, `xsltproc` and their dependencies are installed:
 
 ```bash
-apt-get install cmake make libopenscap8 libxml2-utils ninja-build python3-jinja2 python3-yaml python3-setuptools xsltproc
+apt-get install cmake make libopenscap8 libxml2-utils ninja-build xsltproc
 ```
 
 IMPORTANT: Version `1.0.8` or later of `openscap-utils` is required to build the content.
 
+### Python Dependencies
+
+Some python dependencies must be installed through pip because they're not
+packaged by a distribution.
+
+Dependencies for developing are in `requirements.txt` and you can install them using:
+
+```bash
+$ pip3 install -r requirements.txt
+```
+
+Test dependencies are kept in a separate requirements file called
+`test-requirements.txt` and are installed using the same process:
+
+```bash
+$ pip3 install -r test-requirements.txt
+```
+
 ### Git (Clone the Repository)
 
 Install git if you want to clone the GitHub repository to get the
@@ -92,11 +108,9 @@ apt-get install bats
 
 ### xmldiff (Python unit tests)
 
-Install the  `xmldiff` and `lxml` packages to execute Python unit tests that use these packages.
+Install the `lxml` package to execute Python unit tests that use these packages.
 
 ```bash
-pip3 install xmldiff
-
 # Fedora/RHEL
 yum install python3-lxml
 
@@ -120,11 +134,9 @@ apt-get install yamllint ansible-lint
 
 ### Static Ansible Playbooks tests
 
-Install `yamlpath` and `pytest` to run tests cases that analyse the Ansible
+Install `pytest` to run tests cases that analyse the Ansible
 Playbooks' yaml nodes.
 ```bash
-pip3 install yamlpath
-
 # Fedora/RHEL
 yum install python3-pytest
 
@@ -145,14 +157,6 @@ yum install ninja-build
 apt-get install ninja-build
 ```
 
-### json2html (HTML Report Statistics)
-
-Install the `json2html` package if you want to generate HTML report statistics:
-
-```bash
-pip install json2html
-```
-
 ### Sphinx packages (Developer Documentation)
 
 Install Sphinx packages if you want to generate HTML Documentation, from source directory run:
@@ -211,17 +215,6 @@ yum install python3-mypy
 apt-get install python3-mypy
 ```
 
-#### Type stubs
-
-```bash
-pip install types-openpyxl types-PyYAML openpyxl-stubs
-```
-
-### cmakelint (Linting CMake Files)
-```bash
-pip install cmakelint
-```
-
 ## Downloading the source code
 
 Download and extract a tarball from the [list of releases](https://github.com/ComplianceAsCode/content/releases):

diff --git a/requirements.txt b/requirements.txt
@@ -0,0 +1,14 @@
+cmakelint
+types-openpyxl
+types-PyYAML
+openpyxl-stubs
+openpyxl
+json2html
+pyyaml
+Jinja2
+setuptools
+ruamel.yaml
+pandas
+mypy
+lxml
+pycompliance
diff --git a/test-requirements.txt b/test-requirements.txt
@@ -0,0 +1,4 @@
+yamlpath
+xmldiff
+pytest
+pytest-cov