offchain processing macro and example for sov-nft-module (#939)

* offchain processing * BlockScout integration fixes (#917) * Fixes * retesteth config Signed-off-by: Filippo Costa <[email protected]> --------- Signed-off-by: Filippo Costa <[email protected]> * modules_api: simplify public key definition (#918) * simplify publi key definition * private key * `EVM`: Add missing docs (#914) * rename pending_block to block_env * make evm internals private * refactor query.rs * AccountData doc * query.rs docs * fix exports * add missing docs * cargo fmt * Add readme * Update README.md * Update README.md * better docs * Cleanup prover docs/logs (#919) * Cleanup prever docs/logs * fmt * Bump markdown from 1.0.0-alpha.13 to 1.0.0-alpha.14 (#924) Bumps [markdown](https://github.com/wooorm/markdown-rs) from 1.0.0-alpha.13 to 1.0.0-alpha.14. - [Release notes](https://github.com/wooorm/markdown-rs/releases) - [Commits](wooorm/markdown-rs@1.0.0-alpha.13...1.0.0-alpha.14) --- updated-dependencies: - dependency-name: markdown dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump rayon from 1.7.0 to 1.8.0 (#923) Bumps [rayon](https://github.com/rayon-rs/rayon) from 1.7.0 to 1.8.0. - [Changelog](https://github.com/rayon-rs/rayon/blob/master/RELEASES.md) - [Commits](rayon-rs/rayon@rayon-core-v1.7.0...rayon-core-v1.8.0) --- updated-dependencies: - dependency-name: rayon dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump prost-build from 0.11.9 to 0.12.1 (#930) Bumps [prost-build](https://github.com/tokio-rs/prost) from 0.11.9 to 0.12.1. - [Release notes](https://github.com/tokio-rs/prost/releases) - [Commits](tokio-rs/prost@v0.11.9...v0.12.1) --- updated-dependencies: - dependency-name: prost-build dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump parking_lot from 0.11.2 to 0.12.1 (#927) Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.11.2 to 0.12.1. - [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md) - [Commits](Amanieu/parking_lot@0.11.2...0.12.1) --- updated-dependencies: - dependency-name: parking_lot dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat: use json as manifest constants instead of toml (#922) * feat: use json as manifest constants instead of toml Currently, we parse the manifest file as TOML. However, we will use JSON for the genesis format. For improved consistency, it is desirable to have the constants manifest file with the same format. * fix CI lints * use parent as argument for error reporting * refactor gas config parse to return declaration * fix ci lints * Bump tungstenite from 0.20.0 to 0.20.1 (#931) Bumps [tungstenite](https://github.com/snapview/tungstenite-rs) from 0.20.0 to 0.20.1. - [Changelog](https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md) - [Commits](snapview/tungstenite-rs@v0.20.0...v0.20.1) --- updated-dependencies: - dependency-name: tungstenite dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * EVM: Implement eth_call (#921) * implement eth_call * implement tests and error handling for eth_call * move errors inside evm crate * cleanup result * fix call env * comment test differences * rebase * Revert "rebase" This reverts commit 44e41b2. * fix unused imports * fix review notes * use mix_hash as prevrandao * improve test s * cargo lock * some fixes * move common code into lib * added issue to the nft script * add nft cli binary to nft-utils * Revert "add nft cli binary to nft-utils" This reverts commit d443009. * move nft-utils back to sovereign/nft-utils and move sql to sql.rs * remove un-necessary deps in nft-utils * Run tests with prover enabled. (#932) * Remove demo-rollup dep from demo-prover * add methods to demo-rollup * fix cargo hack * fix cargo hack * Cargo.toml * install toolchain * install toolchain * modify workflow * update cargo.toml * rust.yaml * Add guest-mock * use real prover in bank tests * native * fix lint * update yaml * fic coverage * ELF in CI * update bench get_guest_options() * update build.rs * remove local from demo-rollup * fix lint * remove ide_setup * Remove riscv32im-risc0-zkvm-elf * fix: require `native` when applicable for all targets (#937) * fix: require `native` when applicable for all targets Prior to this commit, some checks of all-targets without the feature `native` would break. This commit introduces a fix for every workspace member to be consistent with the feature set. * change feature requirement to self-dev-dep * update deps * fix dupl celestia dependencies * Remove default trait bound from Module (#941) * Remove demo-rollup-local job from CI (#943) * Delta's go first, remove `get` from StateCheckpoint (#953) * Make genesis config serializable (#956) * Make genesis config serializable * Evm config serde * Fix chain state integ test * fix vec-setter * Introduce: `PublicKeyHex` in `sov-modules-api` (#954) * Add PublicKeyHex * PubKeyHex impl * Update DefaultPublicKey::from_str * Add tests * Remove println * Add doc * estimate-gas signed (#947) * remove include and make it a module * fix packages.yml * EVM: Implement account related RPC (#958) * implement account endpoints * test account endpoints * remove unnecessary conversions * feature: gas meter (#795) * feat: add gas meter to working set This commit introduces `GasMeter`, encapsulated by `WorkingSet`. It will allow the user to consume scalar gas from the working set, and define arbitrary price parsed from a constants.json manifest file at compilation. At each compilation, the `ModuleInfo` derive macro will parse such file, and set the gas price configuration. * fix lint fmt * fix ci test expected error string * update default context to 2 dimensions * Read `accounts` genesis from a file. (#959) * PrivateKeyHex in accounts * Add test_config_serialization * Add accounts.json * fix ci * fix CI * cleanup * Arbitrary PublicKeyHex * PublicKeyHex impl * fix a wrong merge conflict --------- Signed-off-by: Filippo Costa <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Filippo Neysofu Costa <[email protected]> Co-authored-by: Blazej Kolad <[email protected]> Co-authored-by: Preston Evans <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Victor Lopes <[email protected]> Co-authored-by: Orkun Mahir Kılıç <[email protected]> Co-authored-by: Nikolai Golub <[email protected]>
Sovereign-Labs · Sep 29, 2023 · 47650f1 · 47650f1
1 parent 7229226
commit 47650f1
Show file tree

Hide file tree

Showing 22 changed files with 850 additions and 8 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -18,7 +18,9 @@ members = [
 
     "utils/zk-cycle-macros",
     "utils/zk-cycle-utils",
+    "utils/nft-utils",
     "utils/bashtestmd",
+    "utils/nft-utils",
 
     "module-system/sov-cli",
     "module-system/sov-modules-stf-template",

diff --git a/examples/demo-prover/methods/guest-celestia/Cargo.lock b/examples/demo-prover/methods/guest-celestia/Cargo.lock
diff --git a/examples/demo-prover/methods/guest-mock/Cargo.lock b/examples/demo-prover/methods/guest-mock/Cargo.lock
diff --git a/examples/demo-rollup/Cargo.toml b/examples/demo-rollup/Cargo.toml
@@ -79,7 +79,7 @@ native = ["anyhow", "jsonrpsee", "serde", "serde_json", "tracing", "tokio", "tra
 	"sov-state/native", "sov-cli", "clap", "sov-celestia-adapter/native", "sov-db", "sov-sequencer", "sov-stf-runner/native",
 	"sov-modules-api/native", "sov-rollup-interface/native"]
 bench = ["native", "async-trait", "borsh", "hex"]
-
+offchain = ["demo-stf/offchain"]
 
 [[bench]]
 name = "rollup_bench"

diff --git a/examples/demo-stf/Cargo.toml b/examples/demo-stf/Cargo.toml
@@ -51,6 +51,7 @@ rand = "0.8"
 
 [features]
 default = []
+offchain = ["sov-nft-module/offchain"]
 experimental = ["sov-evm/experimental", "reth-primitives"]
 native = [
     "sov-stf-runner/native",

diff --git a/full-node/sov-sequencer/src/utils.rs b/full-node/sov-sequencer/src/utils.rs
@@ -38,6 +38,39 @@ impl SimpleClient {
         Ok(())
     }
 
+    /// Sends multiple transactions to the sequencer for immediate publication.
+    pub async fn send_transactions<Tx: BorshSerialize>(
+        &self,
+        txs: Vec<Tx>,
+        chunk_size: Option<usize>,
+    ) -> Result<(), anyhow::Error> {
+        let serialized_txs: Vec<Vec<u8>> = txs
+            .into_iter()
+            .map(|tx| tx.try_to_vec())
+            .collect::<Result<_, _>>()?;
+
+        match chunk_size {
+            Some(batch_size) => {
+                for chunk in serialized_txs.chunks(batch_size) {
+                    let response: String = self
+                        .http_client
+                        .request("sequencer_publishBatch", chunk.to_vec())
+                        .await?;
+                    info!("publish batch response for chunk: {:?}", response);
+                }
+            }
+            None => {
+                let response: String = self
+                    .http_client
+                    .request("sequencer_publishBatch", serialized_txs)
+                    .await?;
+                info!("publish batch response: {:?}", response);
+            }
+        }
+
+        Ok(())
+    }
+
     /// Get a reference to the underlying [`HttpClient`]
     pub fn http(&self) -> &HttpClient {
         &self.http_client

diff --git a/full-node/sov-stf-runner/src/batch_builder.rs b/full-node/sov-stf-runner/src/batch_builder.rs
@@ -125,14 +125,14 @@ where
             current_batch_size += tx_len;
         }
 
-        if txs.is_empty() {
-            bail!("No valid transactions are available");
-        }
-
         for (tx, err) in dismissed {
             warn!("Transaction 0x{} was dismissed: {:?}", hex::encode(tx), err);
         }
 
+        if txs.is_empty() {
+            bail!("No valid transactions are available");
+        }
+
         Ok(txs)
     }
 }

diff --git a/module-system/module-implementations/sov-nft-module/Cargo.toml b/module-system/module-implementations/sov-nft-module/Cargo.toml
@@ -1,5 +1,6 @@
 [package]
 name = "sov-nft-module"
+description = "A Sovereign SDK module for managing non-fungible tokens"
 version = { workspace = true }
 edition = { workspace = true }
 authors = { workspace = true }
@@ -21,6 +22,10 @@ sov-modules-api = { path = "../../sov-modules-api" }
 sov-modules-macros = {path = "../../sov-modules-macros"}
 sov-state = { path = "../../sov-state" }
 
+postgres = { version = "0.19.7", optional = true }
+tokio = { version = "1.32.0", features=["full"], optional = true }
+tracing = { workspace = true, optional = true }
+
 [dev-dependencies]
 sov-rollup-interface = { path = "../../../rollup-interface" }
 sov-data-generators = { path = "../../utils/sov-data-generators" }
@@ -29,5 +34,6 @@ sov-nft-module = { version = "*", features = ["native"], path = "." }
 
 [features]
 default = []
+offchain = ["postgres","tokio","tracing"]
 native = ["serde_json", "jsonrpsee", "schemars", "sov-state/native", "sov-modules-api/native", ]
 test = ["native"]
diff --git a/module-system/module-implementations/sov-nft-module/offchain_readme.md b/module-system/module-implementations/sov-nft-module/offchain_readme.md
@@ -0,0 +1,60 @@
+## Offchain testing
+
+### Introduction
+This readme outlines the steps to demonstrate the offchain processing functionality that is part of the `sov-nft-module`
+
+### Steps
+* Install postgres on your system
+* Start the postgres terminal
+* Create the tables necessary for offchain processing
+
+```bash
+psql postgres -f sovereign/module-system/module-implementations/sov-nft-module/src/init_db.sql
+```
+* The above command runs the `init_db.sql` script which creates 3 tables
+  * `collections` - tracking the NFT collections that have been created, their supply and other info
+  * `nfts` - tracking the individual NFTs, the `token_uri` pointing to offchain state and other info
+    * `top_owners` - tracks the number of NFTs of each collection that a user owns
+      * running the following query can show the top owners for a specific collection
+      ```sql
+      SELECT owner, count 
+        FROM top_owners
+        WHERE collection_address = (
+          SELECT collection_address
+          FROM collections
+          WHERE collection_name = 'your_collection_name'
+        )
+        ORDER BY count DESC
+        LIMIT 5;
+      ```
+      * running the following query can show the largest owner for each collection
+      ```sql
+      SELECT
+        collection_name,
+        owner,
+        count
+      FROM (
+          SELECT c.collection_name, t.owner, t.count,
+          RANK() OVER (PARTITION BY t.collection_address ORDER BY t.count DESC) as rank
+          FROM top_owners t
+          INNER JOIN collections c ON c.collection_address = t.collection_address
+      ) sub
+      WHERE rank = 1;
+      ```        
+
+* Run the demo rollup in offchain mode
+```bash
+rm -rf demo_data; POSTGRES_CONNECTION_STRING="postgresql://username:password@localhost/postgres" cargo run --features offchain -- --da-layer mock
+```
+* Explanation of the above command
+  * `rm -rf demo_data` is to wipe the rollup state. For testing its better to start with clean state
+  * `POSTGRES_CONNECTION_STRING` is to allow the offchain component of the `sov-nft-module` to connect to postgres instance
+  * `--features offchain` is necessary to enable offchain processing. Without the feature, the functions are no-ops
+  * `--da-layer mock` is used to run an in-memory local DA layer
+* Run the NFT minting script
+```bash
+$ cd sovereign/utils/nft-utils
+$ cargo run
+```
+  * The above script creates 3 NFT collections, mints some NFTs to each collection
+  * The tables can be explored by connecting to postgres and running sample queries from above