[bugfix] Fix possible deadlock in PBS-based scheduler backends when a job is cancelled immediately after submission

[teojgo]

• Create the stdout, stderr files if the don't exist
  to make the torque scheduler treat the job as finished.

Fixes #1298

[vkarak]

I think the problem is more fundamental, because this deadlock could happen in normal operation. I guess that a better solution would be to fix that in the PBS/Torque backends, such that if the job is cancelled, we should not do the additional check for its stdout/stderr files to mark it as done.

[teojgo]

I think the problem is more fundamental, because this deadlock could happen in normal operation. I guess that a better solution would be to fix that in the PBS/Torque backends, such that if the job is cancelled, we should not do the additional check for its stdout/stderr files to mark it as done.

Then maybe open a new issue to find a fundamental solution, while temporarily have this one so that it does not block the CI?

[vkarak]

Then maybe open a new issue to find a fundamental solution, while temporarily have this one so that it does not block the CI?

No, the solution is quite easy. Set an is_cancelling flag that will be set when cancel() is called and the finished() should simply do the additional check only if this flag is not set.

[vkarak]

lgtm