Implementer's guide: downward messages and HRMP, take 2

roadmap/implementers-guide/src/glossary.md

@@ -7,8 +7,10 @@ Here you can find definitions of a bunch of jargon, usually specific to the Polk
 - Backed Candidate: A Backable Candidate noted in a relay-chain block
 - Backing: A set of statements proving that a Parachain Candidate is backable.
 - Collator: A node who generates Proofs-of-Validity (PoV) for blocks of a specific parachain.
+- DMP: (Downward Message Passing). Message passing from the relay-chain to a parachain.
 - Extrinsic: An element of a relay-chain block which triggers a specific entry-point of a runtime module with given arguments.
 - GRANDPA: (Ghost-based Recursive ANcestor Deriving Prefix Agreement). The algorithm validators use to guarantee finality of the Relay Chain.
+- HRMP: (Horizontally Relay-routed Message Passing). A mechanism for message passing between parachains (hence horizontal) that leverages the relay-chain storage. Predates XCMP.
 - Inclusion Pipeline: The set of steps taken to carry a Parachain Candidate from authoring, to backing, to availability and full inclusion in an active fork of its parachain.
 - Module: A component of the Runtime logic, encapsulating storage, routines, and entry-points.
 - Module Entry Point: A recipient of new information presented to the Runtime. This may trigger routines.
@@ -26,8 +28,11 @@ Here you can find definitions of a bunch of jargon, usually specific to the Polk
 - Runtime API: A means for the node-side behavior to access structured information based on the state of a fork of the blockchain.
 - Secondary Checker: A validator who has been randomly selected to perform secondary approval checks on a parablock which is pending approval.
 - Subsystem: A long-running task which is responsible for carrying out a particular category of work.
+- UMP: (Upward Message Passing) A vertical message passing mechanism from a parachain to the relay chain.
 - Validator: Specially-selected node in the network who is responsible for validating parachain blocks and issuing attestations about their validity.
 - Validation Function: A piece of Wasm code that describes the state-transition function of a parachain.
+- VMP: (Vertical Message Passing) A family of mechanisms that are responsible for message exchange between the relay chain and parachains.
+- XCMP (Cross-Chain Message Passing) A type of horizontal message passing (i.e. between parachains) that allows secure message passing directly between parachains and has minimal resource requirements from the relay chain, thus highly scalable.
 
 Also of use is the [Substrate Glossary](https://substrate.dev/docs/en/knowledgebase/getting-started/glossary).
 

roadmap/implementers-guide/src/runtime/inclusion.md

@@ -68,13 +68,20 @@ All failed checks should lead to an unrecoverable error making the block invalid
   1. Check the collator's signature on the candidate data.
   1. check the backing of the candidate using the signatures and the bitfields, comparing against the validators assigned to the groups, fetched with the `group_validators` lookup.
   1. check that the upward messages, when combined with the existing queue size, are not exceeding `config.max_upward_queue_count` and `config.watermark_upward_queue_size` parameters.
+  1. call `Router::check_processed_downward_messages(para, commitments.processed_downward_messages)` to check that the DMQ is properly drained.
+  1. call `Router::check_hrmp_watermark(para, commitments.hrmp_watermark)` for each candidate to check rules of processing the HRMP watermark.
+  1. check that in the commitments of each candidate the horizontal messages are sorted by ascending recipient ParaId and there is no two horizontal messages have the same recipient.
+  1. using `Router::verify_outbound_hrmp(sender, commitments.horizontal_messages)` ensure that the each candidate's para correctly send horizontal messages.
   1. create an entry in the `PendingAvailability` map for each backed candidate with a blank `availability_votes` bitfield.
   1. create a corresponding entry in the `PendingAvailabilityCommitments` with the commitments.
   1. Return a `Vec<CoreIndex>` of all scheduled cores of the list of passed assignments that a candidate was successfully backed for, sorted ascending by CoreIndex.
 * `enact_candidate(relay_parent_number: BlockNumber, CommittedCandidateReceipt)`:
   1. If the receipt contains a code upgrade, Call `Paras::schedule_code_upgrade(para_id, code, relay_parent_number + config.validationl_upgrade_delay)`.
     > TODO: Note that this is safe as long as we never enact candidates where the relay parent is across a session boundary. In that case, which we should be careful to avoid with contextual execution, the configuration might have changed and the para may de-sync from the host's understanding of it.
   1. call `Router::queue_upward_messages` for each backed candidate, using the [`UpwardMessage`s](../types/messages.md#upward-message) from the [`CandidateCommitments`](../types/candidate.md#candidate-commitments).
+  1. call `Router::queue_outbound_hrmp` with the para id of the candidate and the list of horizontal messages taken from the commitment,
+  1. call `Router::prune_hrmp` with the para id of the candiate and the candidate's `hrmp_watermark`.
+  1. call `Router::prune_dmq` with the para id of the candidate and the candidate's `processed_downward_messages`.
   1. Call `Paras::note_new_head` using the `HeadData` from the receipt and `relay_parent_number`.
 * `collect_pending`:
 

roadmap/implementers-guide/src/runtime/paras.md

@@ -93,7 +93,9 @@ OutgoingParas: Vec<ParaId>;
 
 ## Session Change
 
-1. Clean up outgoing paras. This means removing the entries under `Heads`, `ValidationCode`, `FutureCodeUpgrades`, and `FutureCode`. An according entry should be added to `PastCode`, `PastCodeMeta`, and `PastCodePruning` using the outgoing `ParaId` and removed `ValidationCode` value. This is because any outdated validation code must remain available on-chain for a determined amount of blocks, and validation code outdated by de-registering the para is still subject to that invariant.
+1. Clean up outgoing paras.
+	1. This means removing the entries under `Heads`, `ValidationCode`, `FutureCodeUpgrades`, and `FutureCode`. An according entry should be added to `PastCode`, `PastCodeMeta`, and `PastCodePruning` using the outgoing `ParaId` and removed `ValidationCode` value. This is because any outdated validation code must remain available on-chain for a determined amount of blocks, and validation code outdated by de-registering the para is still subject to that invariant.
+	1. Call `Router::offboard_para(P)` for each offboarded para `P`.
 1. Apply all incoming paras by initializing the `Heads` and `ValidationCode` using the genesis parameters.
 1. Amend the `Parachains` list to reflect changes in registered parachains.
 1. Amend the `Parathreads` set to reflect changes in registered parathreads.

roadmap/implementers-guide/src/runtime/router.md

@@ -1,8 +1,6 @@
 # Router Module
 
-The Router module is responsible for storing and dispatching Upward and Downward messages from and to parachains respectively. It is intended to later handle the XCMP logic as well.
-
-For each enacted block the `queue_upward_messages` entry-point is called.
+The Router module is responsible for all messaging mechanisms supported between paras and the relay chain, specifically: UMP, DMP, HRMP and later XCMP.
 
 ## Storage
 
@@ -19,6 +17,91 @@ RelayDispatchQueues: map ParaId => Vec<UpwardMessage>;
 RelayDispatchQueueSize: map ParaId => (u32, u32);
 /// The ordered list of `ParaId`s that have a `RelayDispatchQueue` entry.
 NeedsDispatch: Vec<ParaId>;
+/// The downward messages addressed for a certain para.
+DownwardMessageQueues: map ParaId => Vec<DownwardMessage>;
+```
+
+### HRMP
+
+HRMP related structs:
+
+```rust,ignore
+/// A type used to designate a HRMP channel between a (sender, recipient).
+type HrmpChannelId = (ParaId, ParaId);
+
+/// A description of a request to open an HRMP channel.
+struct HrmpOpenChannelRequest {
+	/// The sender and the initiator of this request.
+    sender: ParaId,
+	/// The recipient of the opened request.
+    recipient: ParaId,
+	/// Indicates if this request was confirmed by the recipient.
+    confirmed: bool,
+	/// How many session boundaries this request has seen.
+    age: u32,
+	/// The amount that the sender supplied at the time of creation of this request.
+    sender_deposit: Balance,
+	/// The maximum number of messages that can be pending in the channel at once.
+    limit_used_places: u32,
+	/// The maximum total size of the messages that can be pending in the channel at once.
+    limit_used_bytes: u32,
+}
+
+/// A description of a request to close an opened HRMP channel.
+struct HrmpCloseChannelRequest {
+	/// The para which initiated closing an existing channel.
+    /// invariant: equals either to sender or recipient.
+    initiator: ParaId,
+	/// The identifier of an HRMP channel to be closed.
+    id: HrmpChannelId,
+}
+
+/// A metadata of an HRMP channel.
+struct HrmpChannel {
+    /// The amount that the sender supplied as a deposit when opening this channel.
+    sender_deposit: Balance,
+	/// The amount that the recipient supplied as a deposit when accepting opening this channel.
+    recipient_deposit: Balance,
+	/// The maximum number of messages that can be pending in the channel at once.
+    limit_used_places: u32,
+	/// The maximum total size of the messages that can be pending in the channel at once.
+    limit_used_bytes: u32,
+    /// The current number of messages pending in the channel.
+	/// Invariant: should be less or equal to `limit_used_places`.
+    used_places: u32,
+	/// The total size in bytes of all message payloads in the channel.
+	/// Invariant: should be less or equal to `limit_used_bytes`.
+    used_bytes: u32,
+}
+```
+HRMP related storage layout
+
+```rust,ignore
+/// Pending HRMP open channel requests.
+HrmpOpenChannelRequests: Vec<HrmpOpenChannelRequest>;
+/// This mapping tracks how many open channel requests are inititated by a given sender para.
+/// Invariant: `HrmpOpenChannelRequests` should contain the same number of items that has `_.sender == X`
+/// as the number of `HrmpOpenChannelRequestCount` for `X`.
+HrmpOpenChannelRequestCount: map ParaId => u32;
+/// Pending HRMP close channel requests.
+HrmpCloseChannelRequests: Vec<HrmpCloseChannelRequest>;
+/// The HRMP watermark associated with each para.
+HrmpWatermarks: map ParaId => Option<BlockNumber>;
+/// HRMP channel data associated with each para.
+HrmpChannels: map HrmpChannelId => Option<HrmpChannel>;
+/// The indexes that map all senders to their recievers and vise versa.
+/// Invariants:
+/// - for each ingress index entry for `P` each item `I` in the index should present in `HrmpChannels` as `(I, P)`.
+/// - for each egress index entry for `P` each item `E` in the index should present in `HrmpChannels` as `(P, E)`.
+/// - there should be no other dangling channels in `HrmpChannels`.
+HrmpIngressChannelsIndex: map ParaId => Vec<ParaId>;
+HrmpEgressChannelsIndex: map ParaId => Vec<ParaId>;
+/// Storage for the messages for each channel.
+/// Invariant: cannot be non-empty if the corresponding channel in `HrmpChannels` is `None`.
+HrmpChannelContents: map HrmpChannelId => Vec<InboundHrmpMessage>;
+/// Maintains a mapping that can be used to answer the question:
+/// What paras sent a message at the given block number for a given reciever.
+HrmpChannelDigests: map ParaId => Vec<(BlockNumber, Vec<ParaId>)>;
 ```
 
 ## Initialization
@@ -27,9 +110,108 @@ No initialization routine runs for this module.
 
 ## Routines
 
+The following routines are intended to be invoked by paras' upward messages.
+
+* `init_open_channel(recipient)`:
+  1. Check that the origin of this message is a para. We say that the origin of this message is the `sender`.
+  1. Check that the `sender` is not `recipient`.
+  1. Check that the `recipient` para exists.
+  1. Check that there is no existing open channel request from sender to recipient.
+  1. Check that the sum of the number of already opened HRMP channels by the `sender` (the size of the set found `HrmpEgressChannelsIndex` for `sender`) and the number of open requests by the `sender` (the value from `HrmpOpenChannelRequestCount` for `sender`) doesn't exceed the limit of channels (`config.hrmp_max_parachain_outbound_channels` or `config.hrmp_max_parathread_outbound_channels`) minus 1.
+  1. Reserve the deposit for the `sender` according to `config.hrmp_sender_deposit`
+  1. Add a new entry to `HrmpOpenChannelRequests` and increase `HrmpOpenChannelRequestCount` by 1 for the `sender`.
+      1. Set `sender_deposit` to `config.hrmp_sender_deposit`
+      1. Set `limit_used_places` to `config.hrmp_channel_max_places`
+      1. Set `limit_limit_used_bytes` to `config.hrmp_channel_max_size`
+
+* `accept_open_channel(i)`, `i` - is the index of open channel request:
+  1. Check that the designated open channel request exists
+  1. Check that the request's `recipient` corresponds to the origin of this message.
+  1. Reserve the deposit for the `recipient` according to `config.hrmp_recipient_deposit`
+  1. Set the request's `confirmed` flag to `true`.
+
+* `close_channel(sender, recipient)`:
+  1. Check that the channel between `sender` and `recipient` exists
+  1. Check that the origin of the message is either `sender` or `recipient`
+  1. Check that there is no existing intention to close the channel between `sender` and `recipient`.
+  1. Add a new entry to `HrmpCloseChannelRequests` with initiator set to the origin of this message.
+
+Candidate Acceptance Function:
+
+* `check_processed_downward_messages(P: ParaId, processed_downward_messages)`:
+  1. Checks that `DownwardMessageQueues` for `P` is at least `processed_downward_messages` long.
+  1. Checks that `processed_downward_messages` is at least 1 if `DownwardMessageQueues` for `recipient` is not empty.
+* `check_hrmp_watermark(P: ParaId, new_hrmp_watermark)`:
+  1. `new_hrmp_watermark` should be strictly greater than the value of `HrmpWatermarks` for `P` (if any).
+  1. `new_hrmp_watermark` must not be greater than the context's block number.
+  1. in ``HrmpChannelDigests`` for `P` an entry with the block number equal to `new_hrmp_watermark` should exist.
+* `verify_outbound_hrmp(sender: ParaId, Vec<OutboundHrmpMessage>)`:
+  1. For each horizontal message `M` with the channel `C` identified by `(sender, M.recipient)` check:
+      1. exists
+      1. `M`'s payload size summed with the `C.used_bytes` doesn't exceed a preconfigured limit `C.limit_used_bytes`.
+      1. `C.used_places + 1` doesn't exceed a preconfigured limit `C.limit_used_places`.
+
+Candidate Enactment:
+
+* `queue_outbound_hrmp(sender: ParaId, Vec<OutboundHrmpMessage>)`:
+  1. For each horizontal message `HM` with the channel `C` identified by `(sender, HM.recipient)`:
+    1. Append `HM` into `HrmpChannelContents` that corresponds to `C`.
+    1. Locate or create an entry in ``HrmpChannelDigests`` for `HM.recipient` and append `sender` into the entry's list.
+    1. Increment `C.used_places`
+    1. Increment `C.used_bytes` by `HM`'s payload size
+* `prune_hrmp(recipient, new_hrmp_watermark)`:
+  1. From ``HrmpChannelDigests`` for `recipient` remove all entries up to an entry with block number equal to `new_hrmp_watermark`.
+  1. From the removed digests construct a set of paras that sent new messages within the interval between the old and new watermarks.
+  1. For each channel `C` identified by `(sender, recipient)` for each `sender` coming from the set, prune messages up to the `new_hrmp_watermark`.
+  1. For each pruned message `M` from channel `C`:
+      1. Decrement `C.used_places`
+      1. Decrement `C.used_bytes` by `M`'s payload size.
+  1. Set `HrmpWatermarks` for `P` to be equal to `new_hrmp_watermark`
+* `prune_dmq(P: ParaId, processed_downward_messages)`:
+  1. Remove the first `processed_downward_messages` from the `DownwardMessageQueues` of `P`.
+
 * `queue_upward_messages(ParaId, Vec<UpwardMessage>)`:
   1. Updates `NeedsDispatch`, and enqueues upward messages into `RelayDispatchQueue` and modifies the respective entry in `RelayDispatchQueueSize`.
 
+The routines described below are for use within a session change and called by the `Paras` module.
+
+* `offboard_para(P: ParaId)`:
+  1. Remove all inbound channels of `P`, i.e. `(_, P)`,
+  1. Remove all outbound channels of `P`, i.e. `(P, _)`,
+  1. Remove all `DownwardMessageQueues` of `P`.
+  1. Remove `RelayDispatchQueueSize` of `P`.
+  1. Remove `RelayDispatchQueues` of `P`.
+  - Note that we don't remove the open/close requests since they are gon die out naturally.
+TODO: What happens with the deposits in channels or open requests?
+
+## Session Change
+
+1. For each request `R` in `HrmpOpenChannelRequests`:
+    1. if `R.confirmed = false`:
+        1. increment `R.age` by 1.
+        1. if `R.age` reached a preconfigured time-to-live limit `config.hrmp_open_request_ttl`, then:
+            1. refund `R.sender_deposit` to the sender
+            1. decrement `HrmpOpenChannelRequestCount` for `R.sender` by 1.
+            1. remove `R`
+    2. if `R.confirmed = true`,
+        1. check that `R.sender` and `R.recipient` are not offboarded.
+        1. create a new channel `C` between `(R.sender, R.recipient)`.
+            1. Initialize the `C.sender_deposit` with `R.sender_deposit` and `C.recipient_deposit` with the value found in the configuration `config.hrmp_recipient_deposit`.
+            1. Insert `sender` into the set `HrmpIngressChannelsIndex` for the `recipient`.
+            1. Insert `recipient` into the set `HrmpEgressChannelsIndex` for the `sender`.
+        1. decrement `HrmpOpenChannelRequestCount` for `R.sender` by 1.
+        1. remove `R`
+1. For each request `R` in `HrmpCloseChannelRequests` remove the channel identified by `R.id`, if exists.
+
+To remove a channel `C` identified with a tuple `(sender, recipient)`:
+
+1. Return `C.sender_deposit` to the `sender`.
+1. Return `C.recipient_deposit` to the `recipient`.
+1. Remove `C` from `HrmpChannels`.
+1. Remove `C` from `HrmpChannelContents`.
+1. Remove `recipient` from the set `HrmpEgressChannelsIndex` for `sender`.
+1. Remove `sender` from the set `HrmpIngressChannelsIndex` for `recipient`.
+
 ## Finalization
 
   1. Dispatch queued upward messages from `RelayDispatchQueues` in a FIFO order applying the `config.watermark_upward_queue_size` and `config.max_upward_queue_count` limits.

roadmap/implementers-guide/src/types/candidate.md

@@ -145,6 +145,9 @@ struct LocalValidationData {
 	/// which case the code upgrade should be applied at the end of the signaling
 	/// block.
 	code_upgrade_allowed: Option<BlockNumber>,
+
+	// TODO: Does this look sensible?
+	hrmp_ingress: BTreeMap<ParaId /* sender */, Vec<InboundHrmpMessage>>,
 }
 ```
 
@@ -167,6 +170,8 @@ The execution and validation of parachain or parathread candidates produces a nu
 struct CandidateCommitments {
 	/// Fees paid from the chain to the relay chain validators.
 	fees: Balance,
+	/// Messages directed to other paras routed via the relay chain.
+	horizontal_messages: Vec<OutboundHrmpMessage>,
 	/// Messages destined to be interpreted by the Relay chain itself.
 	upward_messages: Vec<UpwardMessage>,
 	/// The root of a block's erasure encoding Merkle tree.
@@ -175,6 +180,10 @@ struct CandidateCommitments {
 	new_validation_code: Option<ValidationCode>,
 	/// The head-data produced as a result of execution.
 	head_data: HeadData,
+	/// The number of messages processed from the DMQ.
+	processed_downward_messages: u32,
+	/// The mark which specifies the block number up to which all inbound HRMP messages are processed.
+	hrmp_watermark: BlockNumber,
 }
 ```
 
@@ -203,11 +212,17 @@ struct ValidationOutputs {
 	global_validation_data: GlobalValidationData,
 	/// The local validation data.
 	local_validation_data: LocalValidationData,
+	/// Messages directed to other paras routed via the relay chain.
+	horizontal_messages: Vec<OutboundHrmpMessage>,
 	/// Upwards messages to the relay chain.
 	upwards_messages: Vec<UpwardsMessage>,
 	/// Fees paid to the validators of the relay-chain.
 	fees: Balance,
 	/// The new validation code submitted by the execution, if any.
 	new_validation_code: Option<ValidationCode>,
+	/// The number of messages processed from the DMQ.
+	processed_downward_messages: u32,
+	/// The mark which specifies the block number up to which all inbound HRMP messages are processed.
+	hrmp_watermark: BlockNumber,
 }
 ```