The master notifies the worker agent to restart the training process.

[workingloong]

What changes were proposed in this pull request?

The agent waits and restarts training processes if the node need to reset hardware.

Why are the changes needed?

The node can reset the hardware to restore the hardware. The agent should restart the training processes after the hardware is reset.

Does this PR introduce any user-facing change?

Yes. Users can use the feature by dlrover-run --reset-hardware

How was this patch tested?

UT.

[codecov]

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (25a06ed) 76.13% compared to head (4ccf334) 76.23%.

Files	Patch %	Lines
dlrover/python/elastic_agent/master_client.py	62.50%	3 Missing ⚠️
dlrover/python/master/watcher/k8s_watcher.py	86.66%	2 Missing ⚠️
dlrover/python/elastic_agent/torch/training.py	83.33%	1 Missing ⚠️
dlrover/python/master/node/job_manager.py	66.66%	1 Missing ⚠️
dlrover/python/master/servicer.py	88.88%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #775      +/-   ##
==========================================
+ Coverage   76.13%   76.23%   +0.09%     
==========================================
  Files         161      161              
  Lines       13867    13958      +91     
==========================================
+ Hits        10558    10641      +83     
- Misses       3309     3317       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.