Store and compare worker+dataset repo versions #624
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
severo
changed the title
also: test the version
increase version
and remove Worker.is_major_version_lower_than_worker
worker_version and dataset_git_revision
A job is skipped if a cache entry already exists, if it is successful, if the worker version had the same major number and if the dataset git revision was the same.
A job is skipped if a cache entry already exists, if it is successful, if the worker version had the same major number and if the dataset git revision was the same.
it will change the format of the cache reports, by adding the worker_version and dataset_git_revision fields
it makes the logic simpler. The only restriction is that we can have only one started job with the same set of arguments (type, dataset, config, split). It's the job of the worker to check if the job should be computed or not (worker.should_skip_job)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
to implement #545