Feature/auto migration #5
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
July 20, 2022 22:01
If migration policies are used they may require the initial state of the dataset to process node distribution correctly. Depending on the future implications this has, regarding memory footprint for example, this is up to change and represents a step into considering policy restrictions and opportunities.
This allows the tracking of objects over multiple read/write cycles. Usable in `migration` via `ProfileMsg`.
The event was swapped to the actual result, this has now been fixed.
We might entirely avoid the db lock and depend on the DMU with their handler for this. TODO
This commits adds a possible value into cache which contains the previous storage location, to prevent loss of information when evicting objects from cache where their actual objectref might not be known upon calling write back on them.
This allows for multiple tests to rely on the same files without additional checking or depedencies between tests. Problematic add the moment is that some tests are order dependent due to the storage not being nulled. Reading some data results in _some_ cases in a checksum error. This happens on either file mode, indicating some problem which has not been discovered yet.
This extension provides an interface to allow for system chosen storage preferences. All nodes now can have an additional preference given which can upgrade the existing prefrerence based on it's children/keys. In combination with Migration Policies upgrades/downgrades of preferences are possible.
added 14 commits
November 1, 2022 14:26
|
Just finishing up the PR right now, found an error with the tests which needs to be fixed first but then we should be ready to go. |
|
Finally ready to be reviewed. Tests are failing now as expected, which we'll have to fix in following PRs. |
added 6 commits
November 11, 2022 13:37
Removing the option eliminates some error-prone workflows and simplifies the access structure. The closing logic has been modified to check for _strong_ reference to the dataset, and only when no other than the current exists will the dataset be closed for good.
michaelkuhn
approved these changes
Nov 22, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR contains changes to allow for the usage of automated migration policies in dynamic and adaptable manner. Two policies are implemented which are closer described in https://git.spacesnek.rocks/johannes/master-thesis.
Changes
A wide range of the code has been modified with substantial changes located in the user interface and DMU. The new "migration" module contains relevant changes to any decision processes made in the automated migration process.
Additionally, this PR fixes some issues of the manual migration interface of datasets and object stores, as well as allowing for DMU node storage hints.
Also, a number of fixes of the existing code have been made, notably we no longer deadlock when allocation unexpectedly. Some tests have also been modified and extended to ensure the functionality of newly added features.
Suggestions
We might consider adding https://github.com/jwuensche/lfu-cache to the organization. This repository contains a fork of mine, which extends a crate with some necessary features for our use-case.
Open Issues
In the thesis we lay out some issues which currently still exists in the stack which we should consider solving soon.
Completed