Archive states at runtime #498

AlexandraRoatis · 2018-05-18T20:07:56Z

Description

Please include a brief summary of the change that this pull request proposes. Include any relevant motivation and context. List any dependencies required for this change.

Functionality for saving full states to a separate database which prevents the deletion of the keys in the archived states. Together with the already implemented state recovery methods this feature allows maintaining the appearance of disk storage for all state, while also reducing the size of the state database.

Fixes Issue #54.

Note: this PR builds on top of #482.

The PR in Refactor and test journal pruning class #482 should be reviewed and merged first. After which this PR can be rebased to master-pre-merge branch.

Type of change

Insert x into the following checkboxes to confirm (eg. [x]):

Bug fix.
New feature.
Breaking change (a fix or feature that causes existing functionality to not work as expected).
Requires documentation update.

Testing

Please describe the tests you used to validate this pull request. Provide any relevant details for test configurations as well as any instructions to reproduce these results.

Existing suite of unit tests.
Running the kernel with pruning and archiving enabled and checking the existence of archived and current states by executing ./aion.sh --dump-state-size X where X is the height of the blockchain.
Running the kernel with a database containing a side chain and ensuring correct behavior in the different sync modes. Recovery from archived states works correctly when sync walks back to already pruned blocks.

Verification

Insert x into the following checkboxes to confirm (eg. [x]):

I have self-reviewed my own code and conformed to the style guidelines of this project.
New and existing tests pass locally with my changes.
I have added tests for my fix or feature.
I have made appropriate changes to the corresponding documentation.
My code generates no new warnings.
Any dependent changes have been made.

…present in the archive db

…od to check if open database added to data store interface

AlexandraRoatis · 2018-05-22T15:41:08Z

rebased to include changes to #482

… main chain

…uirement

AlexandraRoatis · 2018-05-28T22:10:27Z

After this PR the state pruning mode can be set from the config file before starting the kernel.

<!--Data pruning behavior for the state database. Options: FULL, TOP, SPREAD.-->
<!--FULL: the state is not pruned-->
<!--TOP: the state is kept only for the top K blocks; limits sync to branching only within the stored blocks-->
<!--SPREAD: the state is kept for the top K blocks and at regular block intervals-->
<state-storage>FULL</state-storage>

If the kernel is using a previous database, it's recommended to run the CLI state data reorganization before starting the kernel:

./aion.sh --state [CHOSEN_MODE]

When switching from TOP mode to FULL or SPREAD mode one should run the CLI command to avoid issues if a backward sync occurs due to re-branching beyond stored block states. If this is not done the kernel may encounter errors at run time. A simple kernel restart will solve these issues however.

AionJayT · 2018-05-28T19:00:38Z

modAionImpl/src/org/aion/zero/impl/sync/TaskImportBlocks.java

+            }
+
+            // checking if there are restrictions due to pruning
+            if (batch.size() > 0 && chain.isPruneRestricted(batch.get(0).getNumber())) {


batch.size() > 0 -> !batch.empty()

AionJayT · 2018-05-29T14:19:01Z

modAionImpl/src/org/aion/zero/impl/sync/TaskGetHeaders.java

@@ -94,39 +97,51 @@ public void run() {
        // decide the start block number
        long from = 0;
        int size = 24;
+
+        // depends on the number of blocks going BACKWARD
+        state.setMaxRepeats(128 / size + 1);


can we make the backward blk# as a constant? there is a constant class for store the p2p constants.

AionJayT · 2018-05-29T15:28:20Z

modAionImpl/src/org/aion/zero/impl/sync/TaskImportBlocks.java

+            ImportResult importResult = ImportResult.IMPORTED_NOT_BEST;
+
+            // importing last block in batch to see if we can skip batch
+            if (state != null && state.getMode() == PeerState.Mode.FORWARD && batch.size() > 0) {


use !batch.empty() instead of batch.size() > 0 is better

AionJayT · 2018-05-29T16:07:57Z

modAionImpl/src/org/aion/zero/impl/sync/TaskImportBlocks.java

+                                if (state.getRepeated() >= state.getMaxRepeats()) {
+                                    state.setMode(PeerState.Mode.NORMAL);
+                                    state.setBase(chain.getBestBlock().getNumber());
+                                    state.resetLastHeaderRequest();


from line 114 to 270, a lot of code is duplicated, it increased the code readability also the maintenance difficulty.

AionJayT · 2018-05-29T16:16:35Z

modMcf/src/org/aion/mcf/config/CfgDb.java

+        switch (prune_option) {
+            case TOP:
+                // journal prune only
+                this.prune = new CfgPrune(128);


same here, make 128 as a constant

AionJayT · 2018-05-29T16:17:52Z

modMcf/src/org/aion/mcf/config/CfgDb.java

+                break;
+            case SPREAD:
+                // journal prune with archived states
+                this.prune = new CfgPrune(128, 10000);


Is the default 10K blocks?

AionJayT · 2018-05-29T16:19:14Z

modMcf/src/org/aion/mcf/config/CfgPrune.java

+        this.archived = true;
+        this.archive_rate =
+                _archive_rate > MINIMUM_ARCHIVE_RATE ? _archive_rate : MINIMUM_ARCHIVE_RATE;
+    }


do we need the upbound Value?

I don't think an upper bound is necessary.

AionJayT · 2018-05-29T17:34:40Z

modMcf/src/org/aion/mcf/trie/scan/ExtractToDatabase.java

+
+    @Override
+    public void doOnNode(byte[] hash, Value node) {
+        db.put(hash, dummy_value);


what's the second arg for?

Since only the keys are used at present the second argument is not needed. However, it cannot be null since that means deleting the key. In the future it may be used for more meaningful information, like possibly state snapshots to be used in the lightning sync PoC.

AionJayT · 2018-05-29T17:40:17Z

modMcf/src/org/aion/mcf/trie/TrieImpl.java

+                }
+            }
+        }
+    }


can you do extract method for the scanTreeDiffLoop & scanTreeLoop ?

it's difficult to reuse code meaningfully without unnecessary computations in one of the methods

AlexandraRoatis added the enhancement New feature or request label May 18, 2018

AlexandraRoatis added this to the v0.2.7 - Island Peak milestone May 18, 2018

AlexandraRoatis requested review from qoire, iamyulong and AionJayT May 18, 2018 20:07

AionJayT modified the milestones: v0.2.7 - Island Peak, v0.2.8 - University Peak May 22, 2018

AlexandraRoatis added 8 commits May 22, 2018 11:31

new configuration class and interface for pruning

d7c1f58

using new detailed pruning configuration

e319850

bugfix: method parameters should be generic

cf6e9d5

new data source class that deletes from the current db only keys not …

e40b699

…present in the archive db

refactored journal prune class to take a data store as argument; meth…

471614e

…od to check if open database added to data store interface

building the journal prune on top of the data source with archive

9c5b552

archiving the world state at runtime

8e487b5

bugfix: incorrect variable update

631da9e

AlexandraRoatis force-pushed the db-archive-states branch from 463e403 to 631da9e Compare May 22, 2018 15:40

AlexandraRoatis mentioned this pull request May 22, 2018

Enabling JournalPruneDataSource #478

Merged

10 tasks

AlexandraRoatis added 2 commits May 23, 2018 16:46

bugfix: reseting the root back to its initial value after recovery

2e89e3f

optimizing the sync BACKWARD and FORWARD behavior

f8242db

AlexandraRoatis force-pushed the db-archive-states branch from be9f60e to f8242db Compare May 23, 2018 21:08

AlexandraRoatis added 2 commits May 23, 2018 17:36

fixing config file

cf03e54

bugfix: printing the size of the trie states should only consider the…

47da914

… main chain

AlexandraRoatis mentioned this pull request May 24, 2018

database pruning #54

Closed

AlexandraRoatis force-pushed the db-archive-states branch from 5a0e502 to 843378f Compare May 24, 2018 15:35

additional stop condition for sync FORWARD mode

b42dd50

AlexandraRoatis force-pushed the db-archive-states branch from 843378f to b42dd50 Compare May 24, 2018 15:38

AlexandraRoatis added 3 commits May 24, 2018 16:39

new pruning configurations and default values

27a7a52

correcting broken test cases

ada9502

extracting the buildGenesis functionality to a utils class

0e4880a

state pruning as CLI options

188a483

AlexandraRoatis force-pushed the db-archive-states branch from d9027dc to 188a483 Compare May 25, 2018 21:03

AlexandraRoatis added 4 commits May 25, 2018 17:25

skipping full batch when in FORWARD mode and last block satisfies req…

d87ac87

…uirement

checking for prune restrictions before importing blocks

d415e4a

moved the prune restriction to block filtering

eeb8627

bugfix: checking for null keys, values and batches

8a1160b

AlexandraRoatis force-pushed the db-archive-states branch from 492d4ad to 8a1160b Compare May 28, 2018 18:34

AionJayT requested changes May 29, 2018

View reviewed changes

AlexandraRoatis added 3 commits May 29, 2018 14:51

backward sync step as constant

51edfc4

extracting repeated code into methods in TaskImportBlocks

9e2d789

prune parameters as constants

bff2e1c

iamyulong approved these changes May 30, 2018

View reviewed changes

bugfix: reducing the heap space requirements for the state recovery

e0ca459

AlexandraRoatis force-pushed the db-archive-states branch from 59eca2f to e0ca459 Compare May 30, 2018 21:38

AionJayT approved these changes May 31, 2018

View reviewed changes

AionJayT merged commit 6aa6299 into db-prune-refactor Jun 1, 2018

AionJayT deleted the db-archive-states branch June 1, 2018 17:06

AionJayT restored the db-archive-states branch June 1, 2018 17:06

AlexandraRoatis mentioned this pull request Jun 1, 2018

Archive states & prune refactor #513

Merged

4 tasks

AionJayT mentioned this pull request Jun 14, 2018

merge v0.2.8 features into master #525

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Archive states at runtime #498

Archive states at runtime #498

AlexandraRoatis commented May 18, 2018 •

edited

Loading

AlexandraRoatis commented May 22, 2018

AlexandraRoatis commented May 28, 2018

AionJayT May 28, 2018

AionJayT May 29, 2018

AionJayT May 29, 2018

AionJayT May 29, 2018

AionJayT May 29, 2018

AionJayT May 29, 2018

AionJayT May 29, 2018

AlexandraRoatis May 29, 2018

AionJayT May 29, 2018

AlexandraRoatis May 29, 2018

AionJayT May 29, 2018

AlexandraRoatis May 29, 2018

Archive states at runtime #498

Archive states at runtime #498

Conversation

AlexandraRoatis commented May 18, 2018 • edited Loading

Description

Note: this PR builds on top of #482.

Type of change

Testing

Verification

AlexandraRoatis commented May 22, 2018

AlexandraRoatis commented May 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexandraRoatis commented May 18, 2018 •

edited

Loading