RAFT Add support for resuming raft.py dataset generation in case of interruption
#394
Labels
enhancement
New feature or request
Comments
ShishirPatil
pushed a commit
that referenced
this issue
May 4, 2024
Implemented a "safe"/recovery mode that periodically saves chunks into "checkpoint" datasets while also keeping track of the current chunk number. In the case of an interruption or crash, the script resumes at the saved chunk number. After all chunks have been processed, all checkpoint datasets are concatenated and saved as one final dataset. Added an argument allowing user to choose whether to run RAFT in safe or fast mode (defaults to safe). Close #394 --------- Co-authored-by: Kaihao Wen <[email protected]>
devanshamin
pushed a commit
to devanshamin/gorilla
that referenced
this issue
Jul 9, 2024
Implemented a "safe"/recovery mode that periodically saves chunks into "checkpoint" datasets while also keeping track of the current chunk number. In the case of an interruption or crash, the script resumes at the saved chunk number. After all chunks have been processed, all checkpoint datasets are concatenated and saved as one final dataset. Added an argument allowing user to choose whether to run RAFT in safe or fast mode (defaults to safe). Close ShishirPatil#394 --------- Co-authored-by: Kaihao Wen <[email protected]>
aw632
pushed a commit
to vinaybagade/gorilla
that referenced
this issue
Aug 22, 2024
Implemented a "safe"/recovery mode that periodically saves chunks into "checkpoint" datasets while also keeping track of the current chunk number. In the case of an interruption or crash, the script resumes at the saved chunk number. After all chunks have been processed, all checkpoint datasets are concatenated and saved as one final dataset. Added an argument allowing user to choose whether to run RAFT in safe or fast mode (defaults to safe). Close ShishirPatil#394 --------- Co-authored-by: Kaihao Wen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The
raft.pydataset generation script takes a long time to run and can be interrupted for various reasons (laptop going to sleep, network errors, API unavailable temporarily, ...). In that case, the script needs to be run all over again.The
raft.pyscript should be able to resume progress after an interruption to avoid starting all over again.The text was updated successfully, but these errors were encountered: