jobs: adding job class

[roksys]

• jobManager super class is responsible for job creation,
  execution/submission, deletion/stop and etc. Thic class
  should be inherited by child classes of specific backend
  (K8s, HTCondor, Slurm and etc.) Closes multiple job backend support #118

Co-authored-by: Diego Rodriguez Rodriguez [email protected]
Signed-off-by: Rokas Maciulaitis [email protected]

[tiborsimko]

s/submission/execution/g everywhere? Seems more natural to take the r-j-controller's internal point of view and talk about executing jobs that were "already submitted" to it by the r-w-engine. E.g. thinking of k8s or docker execution scenarios irst. For HPC/HTC scenarios, the jobs will be sent (i.e. submitted) elsewhere indeed, so e.g. from SLURM's point of view there will be another submission, however from the r-j-c's internal point of view this is just a way of executing the jobs it was asked to run; the r-j-c will just select to proxy them to another backend. (In other words, a "primary submission" is happening between r-w-e and r-j-c, and it is only a "secondary submission" that is happening between r-j-c and SLURM for some backends, so to speak.) Hence my preference for speaking about executing rather then submitting jobs inside r-j-c in general.

[tiborsimko]

Note that @lukasheinrich was trying out git-base syncing of needed input/output files instead of rsync-based syncing

[diegodelemos]

Works locally and looks good 👍 we just need to make Travis happy, polish a couple of things and create issues for what needs to be done after:

For now:

• Make Travis build pass, there are a couple of isort problems and we need to amend tests (for instance, the after execution hook is now not called so test will fail).
• I've been checking why we do not use the JobStatus enum when writing to DB, we use the name as a string, and I can't find it so if possible we should use the enum so it is more readable and less error prone.
• Clean up the k8s.py module removing the functionality that has been moved to kubernetes_job_manager.py.
• Polish documentation removing HTCondor and adding maybe a short description of Kubernetes implementation.

For later, to translate into issues:

• Refactoring of the k8s_watch_jobs. It is not clear yet how we are going to implement the other backends, if directly calling from RJC or creating Kubernetes jobs which are capable of running against specific backends. In the case of calling backend directly from RJC, we will need a job watcher per backend (i.e. htcondor_watch_jobs) so this implementation should serve as reference.
• Manage Kubernetes objects as we do in RWC, for example here, with the official library instead of using dicts.
• Plug after_execution_hook to k8s_watch_jobs so we actually execute it after the job is completely done.