Skip to content

Commit

Permalink
Rename workflow_engine to jobflow.
Browse files Browse the repository at this point in the history
  • Loading branch information
Marcelo Ariel Hamra committed Jun 13, 2024
1 parent 1fba0b8 commit 5d9d780
Show file tree
Hide file tree
Showing 61 changed files with 442 additions and 440 deletions.
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ venv
wazuh_testing.egg-info
dist
deployability/modules/build
deployability/modules/workflow_engine.egg-info
deployability/modules/jobflow.egg-info

# Python bytecode files
*.pyc
Expand Down
24 changes: 12 additions & 12 deletions deployability/README.MD
Original file line number Diff line number Diff line change
Expand Up @@ -35,27 +35,27 @@ git checkout {project-branch}
pip3 install -r deployability/deps/requirements.txt
```

4. Install the Workflow engine library and its launcher:
4. Install the JobFlow engine library and its launcher:

While in wazuh-qa:

```
cd modules
pip3 uninstall -y workflow_engine && pip3 install .
pip3 uninstall -y jobflow && pip3 install .
```

5. Test Fixture to Execute

It will be necessary to create a fixture (yaml file) where the infrastructure, provisioning, and tests to be executed will be declared.

> Note: It is possible to find some fixture examples in deployability/modules/workflow_engine/examples/
> Note: It is possible to find some fixture examples in deployability/modules/jobflow/examples/

Example:

```
version: 0.1
description: This workflow is used to test agents' deployment for DDT1 PoC
description: This YAML is used to test agents' deployment for DDT1 PoC
variables:
agents-os:
- linux-ubuntu-22.04-amd64
Expand Down Expand Up @@ -247,13 +247,13 @@ depends-on
Execute the command by referencing the parameters required by the library (launcher).

```
python3 -m workflow_engine {.yaml fixture path}
python3 -m jobflow {.yaml fixture path}
```

Example

```
python3 -m workflow_engine modules/workflow_engine/examples/dtt1-agents-poc.yaml
python3 -m jobflow modules/jobflow/examples/dtt1-agents-poc.yaml
```


Expand All @@ -263,11 +263,11 @@ python3 -m workflow_engine modules/workflow_engine/examples/dtt1-agents-poc.yaml

- Modules

The framework has 4 modules (`allocation`, `provision`, `testing`, `observability`) that must act consecutively, and a 5th module that orchestrates the previous modules (`Workflow`).
The framework has 4 modules (`allocation`, `provision`, `testing`, `observability`) that must act consecutively, and a 5th module that orchestrates the previous modules (`Jobflow`).

|Module|Description
|--|--|
|Workflow (Orchestrator)|Receives a YAML containing all the instructions to execute for the test development. It contains instructions for allocation, provision, and testing.
|JobFlow (Orchestrator)|Receives a YAML containing all the instructions to execute for the test development. It contains instructions for allocation, provision, and testing.
|Allocation| Receives instructions for the desired architecture and creates the structures, generating IPs, and ports either in AWS or locally with Vagrant.
|Provision| Installs applications on the structures created in allocation.
|Testing| Executes tests on the previously defined structures and triggers actions depending on the test.
Expand All @@ -285,7 +285,7 @@ wazuh-qa/
│ ├── generic
│ ├── provision
│ ├── testing
│ └── workflow_engine
│ └── jobflow
└── plugins
```
Deployability contains the following directories:
Expand All @@ -299,18 +299,18 @@ Deployability contains the following directories:

#### Overview

![image](https://github.com/wazuh/wazuh-qa/assets/125690423/d09dfa61-67cb-410d-8677-bd904d5c4d51)
![image](https://github.com/wazuh/wazuh-qa/assets/2949519/29b600cf-44bd-412c-b02f-9b08353babde)

#### Allocator, provision & test process

![image](https://github.com/wazuh/wazuh-qa/assets/125690423/b1e24a43-13d7-4035-a3e5-5ace4667ec10)
![image](https://github.com/wazuh/wazuh-qa/assets/2949519/9338ccc4-2ffa-47d4-b4cc-86c2a8aa5851)



----


[draw-plot.zip](https://github.com/wazuh/wazuh-qa/files/14330528/plot.zip)
[draw-plot.zip](https://github.com/user-attachments/files/15792115/draw-plot.zip)



Expand Down
26 changes: 13 additions & 13 deletions deployability/modules/allocation/README.MD
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ The Allocation module allows you to create and destroy VMs both locally and in A

#### Set the environment

The execution of the allocation is carried out through the Workflow engine library, or by executing them manually through commands.
The execution of the allocation is carried out through the JobFlow engine, or by executing them manually through commands.
Execution can be done from any operating system.

Initially, you have to install the required Python libraries. We recommend using virtual environments. Follow the technical documentation at https://docs.python.org/3/library/venv.html.
Expand Down Expand Up @@ -46,30 +46,30 @@ To use this module, you should use a Debian-based system, we recommend using Ubu

These commands must be available to run on the Host Operating System.

#### Use the Allocation module through the Workflow engine
#### Use the Allocation module through the JobFlow engine

Now, it is possible to use the Worklow engine library to launch the provision module by doing the following steps:

1. Install the Workflow engine library and its launcher:
1. Install the JobFlow engine and its launcher:

While in wazuh-qa:

```bash
cd modules
pip3 uninstall -y workflow_engine && pip3 install .
pip3 uninstall -y jobflow && pip3 install .
```

2. Test Fixture to Execute:

It will be necessary to create a fixture (yaml file) where the infrastructure, provisioning, and tests to be executed will be declared.

>Note: It is possible to find some fixture examples in [deployability/modules/workflow_engine/examples/](../workflow_engine/examples)
>Note: It is possible to find some fixture examples in [deployability/modules/jobflow/examples/](../jobflow/examples)
Example:

```bash
version: 0.1
description: This workflow is used to test agents' deployment for DDT1 PoC
description: This YAML file is used to test agents' deployment for DDT1 PoC
variables:
agent-os:
- linux-ubuntu-18.04-amd64
Expand Down Expand Up @@ -161,19 +161,19 @@ Now, it is possible to use the Worklow engine library to launch the provision mo
Execute the command by referencing the parameters required by the library (launcher).

```bash
python3 -m workflow_engine {.yaml fixture path}
python3 -m jobflow {.yaml fixture path}
```

Example

```bash
python3 -m workflow_engine modules/workflow_engine/examples/dtt1-agents-poc.yaml
python3 -m jobflow modules/jobflow/examples/dtt1-agents-poc.yaml
```


#### Manual execution of the Allocation module

If one wishes to execute the allocaation module without installing the Workflow engine, they can proceed by using the launcher ([module/allocation/main.py](main.py)):
If one wishes to execute the allocaation module without installing the JobFlow engine, they can proceed by using the launcher ([module/allocation/main.py](main.py)):

1. Create

Expand Down Expand Up @@ -318,7 +318,7 @@ If one wishes to execute the allocaation module without installing the Workflow

The allocation module allows creating infrastructure on both AWS and locally (using Vagrant).

Instructions can be initiated from the fixture and executed through the Workflow engine or executed using Python commands.
Instructions can be initiated from the fixture and executed through the JobFlow engine or executed using Python commands.

In either case, the following information will be needed:

Expand Down Expand Up @@ -350,7 +350,7 @@ python3 modules/allocation/main.py --action create --provider vagrant --size lar

#### General-specific functions

- **Launcher** ([/wazuh-qa/deployability/modules/allocation/main.py](main.py)): The entry point for the workflow or the user who wishes to execute a test.
- **Launcher** ([/wazuh-qa/deployability/modules/allocation/main.py](main.py)): The entry point of the module. The command line or the JobFlow engine typically invokes this entry point.

- **Module functions** ([/wazuh-qa/deployability/modules/allocation/allocation.py](allocation.py)): Module-specific functions responsible for triggering the allocation.

Expand All @@ -373,10 +373,10 @@ python3 modules/allocation/main.py --action create --provider vagrant --size lar

#### Diagram

![image](https://github.com/wazuh/wazuh-qa/assets/64099752/d302f96b-70c0-4fbc-869b-914973ee8186)
![image](https://github.com/wazuh/wazuh-qa/assets/2949519/c673b3ec-ac6a-4ca0-b625-9f98be422f20)


[Allocation.drawio.zip](https://github.com/wazuh/wazuh-qa/files/14868775/Allocation.drawio.zip)
[Allocation.drawio.zip](https://github.com/user-attachments/files/15792190/Allocation.drawio.zip)

### Known issues

Expand Down
2 changes: 1 addition & 1 deletion deployability/modules/generic/logger/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ handlers:
class: logging.FileHandler
level: DEBUG
formatter: simple
filename: /tmp/workflow.log
filename: /tmp/jobflow.log
filters: [uppercase]
loggers:
allocator:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
## Workflow engine
## JobFlow engine

### User documentation

The execution of the Workflow is done through the installation of its library.
The execution of the JobFlow engine is done through the installation of its library.

Initially, Python libraries must be installed. It is recommended to use virtual environments. Follow the technical documentation at https://docs.python.org/3/library/venv.html.

Expand All @@ -29,26 +29,26 @@ To use this module, you should use a Debian-based system, we recommend using Ubu
pip3 install -r deployability/deps/requirements.txt
```

4. Install the Workflow engine library and its launcher:
4. Install the JobFlow engine library and its launcher:

While in wazuh-qa:

```bash
cd modules
pip3 uninstall -y workflow_engine && pip3 install .
pip3 uninstall -y jobflow && pip3 install .
```

5. Test Fixture to Execute:

It will be necessary to create a fixture (yaml file) where the infrastructure, provisioning, and tests to be executed will be declared.
It will be necessary to create a fixture (YAML file) where the infrastructure, provisioning, and tests to be executed will be declared.

>Note: It is possible to find some fixture examples in deployability/modules/workflow_engine/examples/
>Note: It is possible to find some fixture examples in deployability/modules/jobflow/examples/
Example:

```bash
version: 0.1
description: This workflow is used to test agents' deployment for DDT1 PoC
description: This YAML is used to test agents' deployment for DDT1 PoC
variables:
agents-os:
- linux-ubuntu-22.04-amd64
Expand Down Expand Up @@ -240,13 +240,13 @@ To use this module, you should use a Debian-based system, we recommend using Ubu
Execute the command by referencing the parameters required by the library (launcher).

```bash
python3 -m workflow_engine {.yaml fixture path}
python3 -m jobflow {.yaml fixture path}
```

Example

```bash
python3 -m workflow_engine modules/workflow_engine/examples/dtt1-agents-poc.yaml
python3 -m jobflow modules/jobflow/examples/dtt1-agents-poc.yaml
```

> Note The command execution can also be mediated through Jenkins.
Expand All @@ -255,12 +255,12 @@ To use this module, you should use a Debian-based system, we recommend using Ubu

### Technical documentation

`Workflow Engine` is the orchestrator of the deployability test architecture.
`JobFlow engine` is the orchestrator of the deployability test architecture.

Its function is to allow the ordered and structured execution in steps of allocation, provision, and testing.

`The Workflow Engine` receives instructions through a `YAML document`, the structure of which can be exemplified in tests found in:
`wazuh-qa/deployability/modules/workflow_engine/examples`
`The JobFlow engine` receives instructions through a `YAML document`, the structure of which can be exemplified in tests found in:
`wazuh-qa/deployability/modules/jobflow/examples`

**In these tests**:
- Tasks: define the steps.
Expand Down Expand Up @@ -303,19 +303,19 @@ tasks:
as: agent
```

These tasks are executed by the `Workflow Engine` launcher installed as workflow_engine library in your virtual environment.
These tasks are executed by the `JobFlow engine` launcher installed as jobflow library in your virtual environment.

This launcher receives the parameters, sets up the test logs, and proceeds with the ordered execution.

The parameters sent from the launcher are processed by deployability/modules/workflow_engine/models.py, which checks the nature of the parameters sent and filters out incorrect parameters.
The parameters sent from the launcher are processed by deployability/modules/jobflow/models.py, which checks the nature of the parameters sent and filters out incorrect parameters.

![image](https://github.com/wazuh/wazuh-qa/assets/125690423/32aa77b7-f294-41ac-af93-db8a084dbad1)
![image](https://github.com/wazuh/wazuh-qa/assets/2949519/ee6231c2-9c97-4d9b-8c7d-af1cdce92d28)

These are then sent to `deployability/modules/workflow_engine/workflow_processor.py`, where using `deployability/modules/schemas`, instructions in YAML are received and the schema of the instructions is checked.
These are then sent to `deployability/modules/jobflow/jobflow_processor.py`, where using `deployability/modules/schemas`, instructions in YAML are received and the schema of the instructions is checked.

The commands are executed in the WorkflowProcessor of the same file, which also handles parallel executions and aborts failed executions.
The commands are executed in the JobFlowProcessor of the same file, which also handles parallel executions and aborts failed executions.

[WF.drawio.zip](https://github.com/wazuh/wazuh-qa/files/14167559/WF.drawio.zip)
[WF.drawio.zip](https://github.com/user-attachments/files/15792275/WF.drawio.zip)


### License
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (C) 2015, Wazuh Inc.
# Created by Wazuh, Inc. <[email protected]>.
# This program is a free software; you can redistribute it and/or modify it under the terms of GPLv2
from .workflow_processor import WorkflowProcessor
from .jobflow_processor import JobFlowProcessor
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@
project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '../..'))
sys.path.append(project_root)

from workflow_engine.workflow_processor import WorkflowProcessor
from workflow_engine.models import InputPayload
from jobflow.jobflow_processor import JobFlowProcessor
from jobflow.models import InputPayload


def parse_arguments() -> argparse.Namespace:
"""Parse command line arguments."""
parser = argparse.ArgumentParser(description='Execute tasks in a workflow.')
parser.add_argument('workflow_file', type=str,help='Path to the workflow file (YAML format).')
parser = argparse.ArgumentParser(description='Execute tasks in a JobFlow.')
parser.add_argument('jobflow_file', type=str,help='Path to the workflow file (YAML format).')
parser.add_argument('--threads', type=int, default=1, required=False, help='Number of threads to use for parallel execution.')
parser.add_argument('--dry-run', action='store_true', required=False, help='Display the plan without executing tasks.')
parser.add_argument('--log-level', choices=['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'], default='INFO',
Expand All @@ -28,7 +28,7 @@ def main() -> None:
"""Main entry point."""
try:
args = parse_arguments()
processor = WorkflowProcessor(**dict(InputPayload(**vars(args))))
processor = JobFlowProcessor(**dict(InputPayload(**vars(args))))
signal.signal(signal.SIGINT, processor.handle_interrupt)
processor.run()
except Exception as e:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ tasks:
- track-output: "{working-dir}/manager-{manager-os}/track.yaml"
- label-termination-date: "1d"
- label-team: "qa"
- instance-name: "manager"
on-error: "abort-all"
cleanup:
this: process
Expand Down Expand Up @@ -57,6 +58,7 @@ tasks:
- track-output: "{working-dir}/agent-{agent}/track.yaml"
- label-termination-date: "1d"
- label-team: "qa"
- instance-name: "agent-instance-{agent}"
on-error: "abort-all"
foreach:
- variable: agent-os
Expand Down Expand Up @@ -86,7 +88,7 @@ tasks:
- component: wazuh-manager
type: assistant
version: 4.7.4
live: False
live: True
depends-on:
- "allocate-manager-{manager-os}"
on-error: "abort-all"
Expand All @@ -108,7 +110,7 @@ tasks:
- component: "agent"
- wazuh-version: "4.7.4"
- wazuh-revision: "40717"
- live: False
- live: True
foreach:
- variable: agent-os
as: agent
Expand Down
Loading

0 comments on commit 5d9d780

Please sign in to comment.