infrastructure: run an instance of RJC together with each workflow engine

[diegodelemos]

We could think about deploying a REANA-Job-Controller along with each independent workflow engine (as a sidecar container) so we would have something like follows:

Pros:

• Rather than having a central job controller, which handles many requests (n job status query every 5 seconds per workflow)
• Crash on job controller would crash the whole REANA, but with this approach there is one per workflow.

Cons:

• Number of DB connections increases per workflow
• Number of open stream connections with Kubernetes API Server increases per workflow

[tiborsimko]

Postponing for the next sprint; the pros/cons may be affected by the decision of how we go about HPC support, either for job layer only or also for workflow execution layer. #71 (comment)

[diegodelemos]

Another consideration we have to take into account is how we supply secrets to our runtime pods.

• In the optics of continuing RJC as it is right now, a server, we need to pass the secrets from the engines to the RJC API.
• If we go for the side car, we would just add the needed secrets to the deployment as we will do with the engines and jobs.

The following diagram illustrates the rough differences:

[diegodelemos]

Closed by reanahub/reana-commons#122, reanahub/reana-workflow-engine-yadage#117, reanahub/reana-workflow-engine-cwl#104, reanahub/reana-workflow-controller#213 and reanahub/reana-workflow-engine-serial#78.