nvidia mismatch recovery

[claraberendsen]

DESCRIPTION

Since moving to cloud instances for the nvidia agents we have encountered issues with versioning resulting in nvml version mismatch errors (See issue). This is an unrecoverable error for the builds, and requires manual steps to bring back the agent to a funtional state.
This PR automates those steps as postbuild actions. If the build failed due to the Nvidia error, we do the following steps:

1. Remove all labels from current agent and add a recovery-process label to indicate that that agent is performing recovery actions, preventing it from taking on any other build.
2. Requeue the job with same parameters with a delay of 70s to give time for the next step.
3. Schedule a system restart on 1 minute (the delay is needed here so that the postbuild action finishes correctly)

TESTS

• Post build actions only run on failed builds
• Post build actions only run on nvidia agents
• Steps run only if regex match
• Addsdummy not actionable labels to agent to not build jobs between current and restart

- [x] Agent is put back online with correct labels after restart.

• Rebuilds job with same parameters
  Manual rebuild: https://build.osrfoundation.org/job/_test_job_from_dsl/80/
  Automatic rebuild after recovery process: https://build.osrfoundation.org/job/_test_job_from_dsl/81/

[claraberendsen]

Build that succeeded after restart

Build that restarted the agent

WIP

• Job fails with agent disconnection (will add a sleep so the build can finish).
• Rebuild job, code commented. Need to test it further.

[claraberendsen]

For documentation purposes to test the log output, I'm manually adding to the test job _test_job_from_dsl a build step that echos the text I'm looking for.
Will do a PR to add this as a param, so we can have a test job that we can inject any necessary text in the logs. It is also possible to add another param, to force a STATUS for the build (e.g. force the build to fail).

[claraberendsen]

@j-rivero would appreciate a review!

[j-rivero]

While reviewing the PR, in the section of passing the parameters to rebuild the failed build I was looking into the naginator plugin that we have in the buildfarm which usually does a really good work in not successful builds just pressing "Retry" on the GUI. The plugin claims to support: "Only rebuild the job if the build's log output contains a given regular expression" which seems to me like it could be the same thing that we are implementing with custom code here. The GUI is:

The custom implementation here has some advantages like restricting it to only relevant gpu nodes on Linux but the use of a plugin can be more reliable that maintaining and debugging custom groovy code. Before I continue with the review, what do you think @claraberendsen , that we can keep going with this PR or move it to use the retry plugin features?

[nuclearsandwich]

Remove all labels from current agent and add a recovery-process label to indicate that that agent is performing recovery actions, preventing it from taking on any other build.

As this PR proceeds I want to raise an alternative tactic we could take and why I think the current action is the preferred one. Instead of dropping all other labels, we could add the recovery-process label and maintain the others. However, doing that without accounting for it everywhere would mean that singleton labels applied to the node being recovered would still show up as present while the node is no longer running jobs. I was slightly fearful of destructive edits to labels but I think it's actually the best thing to do with autorecovery scripts. 👍🏼

[claraberendsen]

As this PR proceeds I want to raise an alternative tactic we could take and why I think the current action is the preferred one. Instead of dropping all other labels, we could add the recovery-process label and maintain the others. However, doing that without accounting for it everywhere would mean that singleton labels applied to the node being recovered would still show up as present while the node is no longer running jobs. I was slightly fearful of destructive edits to labels but I think it's actually the best thing to do with autorecovery scripts. 👍🏼

Also want to point out that the script is resilient to any fail in the steps subsequent to removing all labels. If for any reason the rebuilding of the job fails the catch will add the old labels to the node and report that we couldn't recover. This is to have some atomicity on these "sudo" actions we are performing from the Jenkins master. The only edge case is if Jenkins fails before shuting down command is scheduled we may encounter our agent with recovery-process label and have to manually restart it.

[claraberendsen]

The custom implementation here has some advantages like restricting it to only relevant gpu nodes on Linux but the use of a plugin can be more reliable that maintaining and debugging custom groovy code. Before I continue with the review, what do you think @claraberendsen , that we can keep going with this PR or move it to use the retry plugin features?

I tried the retry plugin and I found this particular issue:

• I could not find the dsl docs on the only re-run if expression is found part. This is the docs I was using as reference.

@j-rivero If you know of a way to use that conditional on dsl I would prefer that strategy over the current one for maintainability. On the custom run only on gpu nodes, with the naginator plugin even though we won't have complete certainty that it will run only on gpu-nvidia nodes, we are not expecting to encounter this log in a normal workflow in any other node. The only case I could see right now is if someone with intent adds that log to a build it would force the rebuild of the job and we won't be able to stop that.

[j-rivero]

I tried the retry plugin and I found this particular issue:
I could not find the dsl docs on the only re-run if expression is found part. This is the docs I was using as reference.
@j-rivero If you know of a way to use that conditional on dsl I would prefer that strategy over the current one for maintainability.

Looks like that this particular feature or part of the API has not being ported to DSL. There is a way I used in the past to inject XML almost directly from the DSL, it is called the configure block. You can see it in action for example in part of the OSRFWinCompilation class.

[claraberendsen]

Post review testing

The testing is limited to the areas modified in the review: rebuild a job when a log is matched and regex matching on postbuild actions.

Tests

• Script executes on agent with gpu-nvidia label #86
• Script doesn't execute on agent with no-gpu-nvidia label #87
• Script doesn't execute on agent with no gpu-nvidia label #88
• Rebuilds job with same parameters only once when error found in log: Failing job #89 -> Rebuilt #90

[j-rivero]

Good to go. Great work @claraberendsen !