Metadata V15: Generate Runtime APIs (#918)

* Update frame-metadata to v15.1.0 Signed-off-by: Alexandru Vasile <[email protected]> * Enable V15 unstable metadata in frame-metadata Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Move validation hashing to dedicated file Signed-off-by: Alexandru Vasile <[email protected]> * Use sp-metadata-ir from substrate to work with metadata Signed-off-by: Alexandru Vasile <[email protected]> * Revert using sp-metadata-ir in favor of conversion to v15 Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Convert v14 to v15 Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Use v15 for validation Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Use v15 for codegen Signed-off-by: Alexandru Vasile <[email protected]> * metadata/bench: Use v15 Signed-off-by: Alexandru Vasile <[email protected]> * Adjust to v15 metadata Signed-off-by: Alexandru Vasile <[email protected]> * Adjust testing Signed-off-by: Alexandru Vasile <[email protected]> * Improve documentation Signed-off-by: Alexandru Vasile <[email protected]> * force CI Signed-off-by: Alexandru Vasile <[email protected]> * rpc: Fetch metadata at version Signed-off-by: Alexandru Vasile <[email protected]> * artifacts: Update polkadot.scale from commit 6dc9e84dde2 Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Fetch V15 using the new API Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Add runtime API interface Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Hash runtime API metadata for validation Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Extract runtime API metadata wrapper from subxt::Metadata Signed-off-by: Alexandru Vasile <[email protected]> * subxt: Adjust hashing cache to reflect root+item keys Signed-off-by: Alexandru Vasile <[email protected]> * rpc: Add raw state_call API method Signed-off-by: Alexandru Vasile <[email protected]> * runtime_api: Add payload with static and dynamic variants Signed-off-by: Alexandru Vasile <[email protected]> * subxt: Allow payloads to call into the runtime Signed-off-by: Alexandru Vasile <[email protected]> * examples: Add example to make a runtime API call both static and dynamic Signed-off-by: Alexandru Vasile <[email protected]> * Update polkadot.rs Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Simplify client fetching Signed-off-by: Alexandru Vasile <[email protected]> * Address feedback and fallback to old API if needed Signed-off-by: Alexandru Vasile <[email protected]> * runtime_api: Make mutability conditional on input params Signed-off-by: Alexandru Vasile <[email protected]> * Regenerate polkadot.rs Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Retain only pallets without runtime API info Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Retry via `Metadata_metadata` without conversion Signed-off-by: Alexandru Vasile <[email protected]> * payload: Remove `Decode` and change validation fn Signed-off-by: Alexandru Vasile <[email protected]> * metadata: Retain runtime API types Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Runtime APIs documentation based on flag Signed-off-by: Alexandru Vasile <[email protected]> * Update examples/examples/custom_metadata_url.rs Co-authored-by: James Wilson <[email protected]> * Update artifacts from polkadot-a6cfdb16e9 Signed-off-by: Alexandru Vasile <[email protected]> * Update polkadot.rs with polkadot-a6cfdb16e9 Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Generate input structures for runtime API Signed-off-by: Alexandru Vasile <[email protected]> * runtime_api: Remove the static paylaod and use single impl Signed-off-by: Alexandru Vasile <[email protected]> * examples: Fetch account nonce Signed-off-by: Alexandru Vasile <[email protected]> * testing: Adjust build script to fetch latest metadata Signed-off-by: Alexandru Vasile <[email protected]> * testing: Check account nonce from runtime API Signed-off-by: Alexandru Vasile <[email protected]> * Update cargo.lock Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Fix doc generation for runtime types Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Rename `inputs` runtime calls module to `types` Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Generate Calls structs inside the types module Signed-off-by: Alexandru Vasile <[email protected]> * testing: Check Alice account nonce before submitting the tx Signed-off-by: Alexandru Vasile <[email protected]> * cli: Add metadata version option flag supporting v14 and unstable Signed-off-by: Alexandru Vasile <[email protected]> * cli: Specify version to fetch Signed-off-by: Alexandru Vasile <[email protected]> * subxt: Fallback to fetching latest stable metadata Signed-off-by: Alexandru Vasile <[email protected]> * subxt: Add unstable-metadata feature to fetch the latest Signed-off-by: Alexandru Vasile <[email protected]> * RuntimeVersion with Latest and Version(u32) Signed-off-by: Alexandru Vasile <[email protected]> * Update polkadot.rs Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Adjust fetch_metadata to inspect version list Signed-off-by: Alexandru Vasile <[email protected]> * testing: Adjust metadata to metadata_legacy Signed-off-by: Alexandru Vasile <[email protected]> * events: Adjust docs to use metadata_legacy Signed-off-by: Alexandru Vasile <[email protected]> * have a pass over fetch_metadata * cargo fmt * Option<String> when fetch metadata via latest API * clippy * fmt * cli: Use the MetadataVersion from codegen Signed-off-by: Alexandru Vasile <[email protected]> * cli: Specify latest as default for MetadataVersion Signed-off-by: Alexandru Vasile <[email protected]> * cli: Remove version from metadata and use the one from file_or_url Signed-off-by: Alexandru Vasile <[email protected]> * Fix clippy Signed-off-by: Alexandru Vasile <[email protected]> * codegen: Decode metadata independently for different RPC calls Signed-off-by: Alexandru Vasile <[email protected]> --------- Signed-off-by: Alexandru Vasile <[email protected]> Co-authored-by: James Wilson <[email protected]>
paritytech · May 4, 2023 · 90699b7 · 90699b7
1 parent fadb039
commit 90699b7
Show file tree

Hide file tree

Showing 33 changed files with 21,694 additions and 16,125 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/artifacts/polkadot_metadata_full.scale b/artifacts/polkadot_metadata_full.scale
diff --git a/cli/src/commands/compatibility.rs b/cli/src/commands/compatibility.rs
@@ -11,6 +11,7 @@ use frame_metadata::{
 use jsonrpsee::client_transport::ws::Uri;
 use serde::{Deserialize, Serialize};
 use std::collections::HashMap;
+use subxt_codegen::utils::MetadataVersion;
 use subxt_metadata::{get_metadata_hash, get_pallet_hash, metadata_v14_to_latest};
 
 /// Verify metadata compatibility between substrate nodes.
@@ -25,16 +26,35 @@ pub struct Opts {
     /// The validation will omit the full metadata check and focus instead on the pallet.
     #[clap(long, value_parser)]
     pallet: Option<String>,
+    /// Specify the metadata version.
+    ///
+    ///  - unstable:
+    ///
+    ///    Use the latest unstable metadata of the node.
+    ///
+    ///  - number
+    ///
+    ///    Use this specific metadata version.
+    ///
+    /// Defaults to latest.
+    #[clap(long = "version", default_value = "latest")]
+    version: MetadataVersion,
 }
 
 pub async fn run(opts: Opts) -> color_eyre::Result<()> {
     match opts.pallet {
-        Some(pallet) => handle_pallet_metadata(opts.nodes.as_slice(), pallet.as_str()).await,
-        None => handle_full_metadata(opts.nodes.as_slice()).await,
+        Some(pallet) => {
+            handle_pallet_metadata(opts.nodes.as_slice(), pallet.as_str(), opts.version).await
+        }
+        None => handle_full_metadata(opts.nodes.as_slice(), opts.version).await,
     }
 }
 
-async fn handle_pallet_metadata(nodes: &[Uri], name: &str) -> color_eyre::Result<()> {
+async fn handle_pallet_metadata(
+    nodes: &[Uri],
+    name: &str,
+    version: MetadataVersion,
+) -> color_eyre::Result<()> {
     #[derive(Serialize, Deserialize, Default)]
     #[serde(rename_all = "camelCase")]
     struct CompatibilityPallet {
@@ -44,7 +64,7 @@ async fn handle_pallet_metadata(nodes: &[Uri], name: &str) -> color_eyre::Result
 
     let mut compatibility: CompatibilityPallet = Default::default();
     for node in nodes.iter() {
-        let metadata = fetch_runtime_metadata(node).await?;
+        let metadata = fetch_runtime_metadata(node, version).await?;
 
         match metadata.pallets.iter().find(|pallet| pallet.name == name) {
             Some(pallet_metadata) => {
@@ -73,10 +93,10 @@ async fn handle_pallet_metadata(nodes: &[Uri], name: &str) -> color_eyre::Result
     Ok(())
 }
 
-async fn handle_full_metadata(nodes: &[Uri]) -> color_eyre::Result<()> {
+async fn handle_full_metadata(nodes: &[Uri], version: MetadataVersion) -> color_eyre::Result<()> {
     let mut compatibility_map: HashMap<String, Vec<String>> = HashMap::new();
     for node in nodes.iter() {
-        let metadata = fetch_runtime_metadata(node).await?;
+        let metadata = fetch_runtime_metadata(node, version).await?;
         let hash = get_metadata_hash(&metadata);
         let hex_hash = hex::encode(hash);
         println!("Node {node:?} has metadata hash {hex_hash:?}",);
@@ -96,8 +116,11 @@ async fn handle_full_metadata(nodes: &[Uri]) -> color_eyre::Result<()> {
     Ok(())
 }
 
-async fn fetch_runtime_metadata(url: &Uri) -> color_eyre::Result<RuntimeMetadataV15> {
-    let bytes = subxt_codegen::utils::fetch_metadata_bytes(url).await?;
+async fn fetch_runtime_metadata(
+    url: &Uri,
+    version: MetadataVersion,
+) -> color_eyre::Result<RuntimeMetadataV15> {
+    let bytes = subxt_codegen::utils::fetch_metadata_bytes(url, version).await?;
 
     let metadata = <RuntimeMetadataPrefixed as Decode>::decode(&mut &bytes[..])?;
     if metadata.0 != META_RESERVED {

diff --git a/cli/src/utils.rs b/cli/src/utils.rs
@@ -5,7 +5,7 @@
 use clap::Args;
 use color_eyre::eyre;
 use std::{fs, io::Read, path::PathBuf};
-use subxt_codegen::utils::Uri;
+use subxt_codegen::utils::{MetadataVersion, Uri};
 
 /// The source of the metadata.
 #[derive(Debug, Args)]
@@ -16,29 +16,57 @@ pub struct FileOrUrl {
     /// The path to the encoded metadata file.
     #[clap(long, value_parser)]
     file: Option<PathBuf>,
+    /// Specify the metadata version.
+    ///
+    ///  - unstable:
+    ///
+    ///    Use the latest unstable metadata of the node.
+    ///
+    ///  - number
+    ///
+    ///    Use this specific metadata version.
+    ///
+    /// Defaults to 14.
+    #[clap(long)]
+    version: Option<MetadataVersion>,
 }
 
 impl FileOrUrl {
     /// Fetch the metadata bytes.
     pub async fn fetch(&self) -> color_eyre::Result<Vec<u8>> {
-        match (&self.file, &self.url) {
+        match (&self.file, &self.url, self.version) {
             // Can't provide both --file and --url
-            (Some(_), Some(_)) => {
+            (Some(_), Some(_), _) => {
                 eyre::bail!("specify one of `--url` or `--file` but not both")
             }
             // Load from --file path
-            (Some(path), None) => {
+            (Some(path), None, None) => {
                 let mut file = fs::File::open(path)?;
                 let mut bytes = Vec::new();
                 file.read_to_end(&mut bytes)?;
                 Ok(bytes)
             }
+            // Cannot load the metadata from the file and specify a version to fetch.
+            (Some(_), None, Some(_)) => {
+                // Note: we could provide the ability to convert between metadata versions
+                // but that would be involved because we'd need to convert
+                // from each metadata to the latest one and from the
+                // latest one to each metadata version. For now, disable the conversion.
+                eyre::bail!("`--file` is incompatible with `--version`")
+            }
             // Fetch from --url
-            (None, Some(uri)) => Ok(subxt_codegen::utils::fetch_metadata_bytes(uri).await?),
+            (None, Some(uri), version) => Ok(subxt_codegen::utils::fetch_metadata_bytes(
+                uri,
+                version.unwrap_or_default(),
+            )
+            .await?),
             // Default if neither is provided; fetch from local url
-            (None, None) => {
+            (None, None, version) => {
                 let uri = Uri::from_static("http://localhost:9933");
-                Ok(subxt_codegen::utils::fetch_metadata_bytes(&uri).await?)
+                Ok(
+                    subxt_codegen::utils::fetch_metadata_bytes(&uri, version.unwrap_or_default())
+                        .await?,
+                )
             }
         }
     }

diff --git a/codegen/src/api/calls.rs b/codegen/src/api/calls.rs
@@ -90,11 +90,11 @@ pub fn generate_calls(
                 pub fn #fn_name(
                     &self,
                     #( #call_fn_args, )*
-                ) -> #crate_path::tx::Payload<#struct_name> {
+                ) -> #crate_path::tx::Payload<types::#struct_name> {
                     #crate_path::tx::Payload::new_static(
                         #pallet_name,
                         #call_name,
-                        #struct_name { #( #call_args, )* },
+                        types::#struct_name { #( #call_args, )* },
                         [#(#call_hash,)*]
                     )
                 }
@@ -120,7 +120,11 @@ pub fn generate_calls(
 
             type DispatchError = #types_mod_ident::sp_runtime::DispatchError;
 
-            #( #call_structs )*
+            pub mod types {
+                use super::#types_mod_ident;
+
+                #( #call_structs )*
+            }
 
             pub struct TransactionApi;
 

diff --git a/codegen/src/api/mod.rs b/codegen/src/api/mod.rs
@@ -8,6 +8,7 @@ mod calls;
 mod constants;
 mod errors;
 mod events;
+mod runtime_apis;
 mod storage;
 
 use frame_metadata::v15::RuntimeMetadataV15;
@@ -18,7 +19,7 @@ use crate::error::CodegenError;
 use crate::{
     ir,
     types::{CompositeDef, CompositeDefFields, TypeGenerator, TypeSubstitutes},
-    utils::{fetch_metadata_bytes_blocking, Uri},
+    utils::{fetch_metadata_bytes_blocking, MetadataVersion, Uri},
     CratePath,
 };
 use codec::Decode;
@@ -95,7 +96,11 @@ pub fn generate_runtime_api_from_url(
     should_gen_docs: bool,
     runtime_types_only: bool,
 ) -> Result<TokenStream2, CodegenError> {
-    let bytes = fetch_metadata_bytes_blocking(url)?;
+    // Fetch latest unstable version, if that fails fall back to the latest stable.
+    let bytes = match fetch_metadata_bytes_blocking(url, MetadataVersion::Unstable) {
+        Ok(bytes) => bytes,
+        Err(_) => fetch_metadata_bytes_blocking(url, MetadataVersion::Latest)?,
+    };
 
     generate_runtime_api_from_bytes(
         item_mod,
@@ -434,6 +439,14 @@ impl RuntimeGenerator {
 
         let rust_items = item_mod_ir.rust_items();
 
+        let apis_mod = runtime_apis::generate_runtime_apis(
+            &self.metadata,
+            &type_gen,
+            types_mod_ident,
+            &crate_path,
+            should_gen_docs,
+        )?;
+
         Ok(quote! {
             #( #item_mod_attrs )*
             #[allow(dead_code, unused_imports, non_camel_case_types)]
@@ -487,6 +500,12 @@ impl RuntimeGenerator {
                     TransactionApi
                 }
 
+                pub fn apis() -> runtime_apis::RuntimeApi {
+                    runtime_apis::RuntimeApi
+                }
+
+                #apis_mod
+
                 pub struct ConstantsApi;
                 impl ConstantsApi {
                     #(