[Morse->Shannon Migration] state export/import - collect accounts #1039

bryanchriswhite · 2025-01-23T13:04:35Z

Summary

Add poktrolld migrate collect-morse-accounts command to convert Morse state export into a MorseAccountState for import into Shannon.

Issue

[Morse->Shannon Migration] Migration module #1034

Type of change

Select one or more from the following:

New feature, functionality or library
Consensus breaking; add the consensus-breaking label if so. See [Infra] Automatically add the consensus-breaking label #791 for details
Bug fix
Code health or cleanup
Documentation
Other (specify)

Testing

Documentation: make docusaurus_start; only needed if you make doc changes
Unit Tests: make go_develop_and_test
LocalNet E2E Tests: make test_e2e
DevNet E2E Tests: Add the devnet-test-e2e label to the PR.

Sanity Checklist

I have tested my changes using the available tooling
I have commented my code
I have performed a self-review of my own code; both comments & source code
I create and reference any new tickets, if applicable
I have left TODOs throughout the codebase, if applicable

bryanchriswhite · 2025-01-27T11:35:21Z

proto/poktroll/migration/types.proto

+    (gogoproto.jsontag) = "jailed",
+    (gogoproto.moretags) = "yaml:\"jailed\""];
+  int32  status = 4 [
+    (gogoproto.jsontag) = "status",
+    (gogoproto.moretags) = "yaml:\"status\""];
+  string staked_tokens = 6;


@Reviewer do we need to consider these fields?

Yes, but add the same qualifiers you have elsewhere. E.g. [(gogoproto.jsontag) = "tokens"];

I think my highlighting was off, I meant to call out jailed and status. If we do need these, what role should they play in the migration?

Regarding the json option, will add. 👍 It needs to be staked_tokens though as this structure is derived from the Morse export data structure (Morse proto types).

cmd/poktrolld/cmd/migrate/migrate_test.go

bryanchriswhite · 2025-01-27T14:28:46Z

cmd/poktrolld/cmd/migrate/migrate.go

+// transformMorseState consolidates the Morse account balance, application stake,
+// and supplier stake for each account as an entry in the resulting MorseAccountState.
+//
+// TODO_IN_THIS_COMMIT: benchmark and consider at what point (i.e. number of accounts) does asynchronous import processing matter?


Seems to bottom out around 10^5 on my machine.

Subsequent observations seem to indicate that performance actually seems to degrade with concurrency.

I took a look at the number of accounts, apps, and suppliers on Morse mainnet; based on this information I don't think it's worthwhile to spend any effort on optimization/parallization here:

object count

accounts 56808

apps 2298

suppliers 13269

$ curl -X POST https://pocket-rpc.liquify.com/v1/query/accounts -d '{"page":1,"per_page":100000}' -o ./morse_accounts.json % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6671k 0 6671k 100 28 2788k 11 0:00:02 0:00:02 --:--:-- 2790k $ jq ".result|length" ./morse_accounts.json 56808 # curl -X POST https://pocket-rpc.liquify.com/v1/query/apps -d '{"opts": {"page":1,"per_page":100000}}' -o ./morse_apps.json % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 600k 0 600k 100 38 1331k 84 --:--:-- --:--:-- --:--:-- 1333k $ jq ".result|length" ./morse_apps.json 2298 $ curl -X POST https://pocket-rpc.liquify.com/v1/query/nodes -d '{"opts": {"page":1,"per_page":100000}}' -o ./morse_suppliers.json % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6172k 0 6172k 100 38 5039k 31 0:00:01 0:00:01 --:--:-- 5043k $ jq ".result|length" ./morse_suppliers.json 13269

Olshansk · 2025-01-29T18:45:21Z

cmd/poktrolld/cmd/migrate/migrate.go

@@ -0,0 +1,191 @@
+package migrate


Given that this is all new code, let's use autocli instead.

AutoCLI does not apply here because there is not gRPC service, message, or query.

The purpose of this command is to facilitate the deterministic (i.e. reproducible) transformation from the Morse export data structure (MorseStateExport) into the Shannon import data structure (MorseAccountState). It does not interact with the network directly.

proto/poktroll/migration/legacy.proto

Olshansk · 2025-01-29T19:03:16Z

proto/poktroll/migration/legacy.proto

@@ -0,0 +1,42 @@
+syntax = "proto3";


Can we rename this to morse.proto?

My rationale here was that we depend on multiple types from Morse that can be divided into two categories:

Those which MUST be persisted on-chain in Shannon

Those which are only required for the export/import state transform CLI command.

Given these categories, both of which I would characterize as "morse". I chose to go with "types.proto" for the on-chain types, for consistency with other modules, and then chose "legacy" for the other types.

Upon further reflection, I've realized that the ONLY types that are needed on-chain in Shannon are MorseAccountState, MorseAccount, and MorsePublicKey. I've moved MorseApplication and MorseValidator to legacy.proto reflect this.

Olshansk · 2025-01-29T19:05:03Z

proto/poktroll/migration/legacy.proto

+message MorseStateExport {
+  string app_hash = 1 [(gogoproto.jsontag) = "app_hash"];
+  MorseAppState app_state = 2 [(gogoproto.jsontag) = "app_state"];
+}


Suggested change

message MorseStateExport {

string app_hash = 1 [(gogoproto.jsontag) = "app_hash"];

MorseAppState app_state = 2 [(gogoproto.jsontag) = "app_state"];

}

// < Provide a description >

// < Prove an example command >

message MorseStateExport {

// app_hash is a hash of ???

string app_hash = 1 [(gogoproto.jsontag) = "app_hash"];

// app_state is < ... >

// It can be retrieved via < ... >

MorseAppState app_state = 2 [(gogoproto.jsontag) = "app_state"];

}

The types need a lot more comments because of how important it is.

E.g. I don't know if app_hash is a hash of app_state or the entire app_hash directly from pocket-core.

MorseStateExport is not a data structure which I designed, it is the result of the pocket util export-genesis-for-reset command.

I will #PUC, but after thinking about it, I think the purpose of app_hash here (which is the Morse tendermint state hash - at the time of export) is to be included in the MorseAccountState#state_hash hash result (i.e. app_state is part of the input), which is used to verify it.

Olshansk · 2025-01-29T19:14:18Z

proto/poktroll/migration/legacy.proto

+message MorseAppState {
+  MorseApplications application = 1 [(gogoproto.jsontag) = "application"];
+  MorseAuth auth = 2 [(gogoproto.jsontag) = "auth"];
+  MorsePos pos = 3 [(gogoproto.jsontag) = "pos"];


#PUC

Is this for validators?

Why not rename to `validators?

Provide examples of how to query it.

I suspect this structure a a result of tendermint convention, I'm assuming that pos is an acronym for "proof of stake".

MorseStateExport is not a data structure which I designed, it is the result of the pocket util export-genesis-for-reset command.

(see: #1039 (comment))

Olshansk · 2025-01-29T21:59:52Z

cmd/poktrolld/cmd/migrate/migrate.go

+
+		coins := exportAccount.Value.Coins
+		if len(coins) == 0 {
+			return nil


Log on this error. Should it ever actually happen?

Olshansk · 2025-01-29T22:00:09Z

cmd/poktrolld/cmd/migrate/migrate.go

+			continue
+		}
+
+		addr := exportAccount.Value.Address.String()


s/addr/morseAddr

Olshansk · 2025-01-29T22:00:31Z

cmd/poktrolld/cmd/migrate/migrate.go

+		}
+
+		// DEV_NOTE: SHOULD ONLY be one denom (upokt).
+		coin := coins[0]


can you assert that it's exactly one?

Olshansk · 2025-01-29T22:07:47Z

cmd/poktrolld/cmd/migrate/migrate.go

+	for _, exportApplication := range inputState.AppState.Application.Applications {
+		addr := exportApplication.Address.String()
+
+		// DEV_NOTE: An account SHOULD exist for each actor.


Olshansk · 2025-01-29T22:07:52Z

cmd/poktrolld/cmd/migrate/migrate.go

+
+		// DEV_NOTE: An account SHOULD exist for each actor.
+		if !morseWorkspace.hasAccount(addr) {
+			// TODO_IN_THIS_COMMIT: consolidate error types...


Let's do it

bryanchriswhite added this to the Beta TestNet Iteration milestone Jan 23, 2025

bryanchriswhite self-assigned this Jan 23, 2025

bryanchriswhite force-pushed the chore/migration/state-prep branch from f648aa5 to e35ca54 Compare January 24, 2025 15:19

bryanchriswhite changed the base branch from scaffold/migration/morse-state to scaffold/migration-module January 27, 2025 11:31

bryanchriswhite force-pushed the chore/migration/state-prep branch from 12a072c to 900c66b Compare January 27, 2025 11:32

bryanchriswhite commented Jan 27, 2025

View reviewed changes

bryanchriswhite force-pushed the chore/migration/state-prep branch 2 times, most recently from 3515058 to 20aa71d Compare January 27, 2025 11:55

bryanchriswhite commented Jan 27, 2025

View reviewed changes

cmd/poktrolld/cmd/migrate/migrate_test.go Outdated Show resolved Hide resolved

bryanchriswhite commented Jan 27, 2025

View reviewed changes

bryanchriswhite linked an issue Jan 27, 2025 that may be closed by this pull request

[Morse->Shannon Migration] Migration module #1034

Open

9 tasks

bryanchriswhite force-pushed the chore/migration/state-prep branch from 08b72b7 to b19a003 Compare January 27, 2025 15:50

bryanchriswhite added 2 commits January 27, 2025 16:52

feat: add migrate collect-morse-account subcommand

85992e2

refactor: test & add benchmark

890d5b5

bryanchriswhite force-pushed the chore/migration/state-prep branch from b19a003 to 890d5b5 Compare January 27, 2025 15:52

fix: linter errors

95e1edf

bryanchriswhite marked this pull request as ready for review January 27, 2025 15:56

bryanchriswhite requested a review from red-0ne January 27, 2025 15:56

fix: add missing files

77bc6a5

bryanchriswhite requested a review from Olshansk January 27, 2025 16:30

bryanchriswhite mentioned this pull request Jan 27, 2025

[Morse->Shannon Migration] Migration module #1034

Open

9 tasks

Olshansk added on-chain On-chain business logic consensus-breaking IMPORTANT! If the PR with this tag is merged, next release WILL HAVE TO BE an upgrade. labels Jan 28, 2025

Olshansk requested changes Jan 29, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Morse->Shannon Migration] state export/import - collect accounts #1039

[Morse->Shannon Migration] state export/import - collect accounts #1039

bryanchriswhite commented Jan 23, 2025 •

edited

Loading

bryanchriswhite Jan 27, 2025

Olshansk Jan 29, 2025

bryanchriswhite Jan 30, 2025

bryanchriswhite Jan 30, 2025

bryanchriswhite Jan 27, 2025 •

edited

Loading

bryanchriswhite Jan 27, 2025 •

edited

Loading

This comment was marked as resolved.

bryanchriswhite Jan 28, 2025 •

edited

Loading

Olshansk Jan 29, 2025

bryanchriswhite Jan 30, 2025 •

edited

Loading

Olshansk Jan 29, 2025

bryanchriswhite Jan 30, 2025

bryanchriswhite Jan 30, 2025

Olshansk Jan 29, 2025

Olshansk Jan 29, 2025

bryanchriswhite Jan 30, 2025 •

edited

Loading

Olshansk Jan 29, 2025

bryanchriswhite Jan 30, 2025 •

edited

Loading

Olshansk Jan 29, 2025

Olshansk Jan 29, 2025

Olshansk Jan 29, 2025

Olshansk Jan 29, 2025

Olshansk Jan 29, 2025

[Morse->Shannon Migration] state export/import - collect accounts #1039

Are you sure you want to change the base?

[Morse->Shannon Migration] state export/import - collect accounts #1039

Conversation

bryanchriswhite commented Jan 23, 2025 • edited Loading

Summary

Issue

Type of change

Testing

Sanity Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryanchriswhite Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

bryanchriswhite Jan 27, 2025 • edited Loading

Choose a reason for hiding this comment

This comment was marked as resolved.

bryanchriswhite Jan 28, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryanchriswhite Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryanchriswhite Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryanchriswhite Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryanchriswhite commented Jan 23, 2025 •

edited

Loading

bryanchriswhite Jan 27, 2025 •

edited

Loading

bryanchriswhite Jan 27, 2025 •

edited

Loading

bryanchriswhite Jan 28, 2025 •

edited

Loading

bryanchriswhite Jan 30, 2025 •

edited

Loading

bryanchriswhite Jan 30, 2025 •

edited

Loading

bryanchriswhite Jan 30, 2025 •

edited

Loading