From 587aacda7149cd6732299711f4913c9b3d6c2b0d Mon Sep 17 00:00:00 2001 From: Age Manning Date: Tue, 17 May 2022 15:22:39 +1000 Subject: [PATCH 1/4] Initial draft of gossipsub v2 --- pubsub/gossipsub/gossipsub-v2.0.md | 80 ++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 pubsub/gossipsub/gossipsub-v2.0.md diff --git a/pubsub/gossipsub/gossipsub-v2.0.md b/pubsub/gossipsub/gossipsub-v2.0.md new file mode 100644 index 000000000..12136b872 --- /dev/null +++ b/pubsub/gossipsub/gossipsub-v2.0.md @@ -0,0 +1,80 @@ +# Episub - Gossipsub Extension + + +# Overview + +This document aims to provide a minimal extension to the [gossipsub v1.1](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md) protocol, that supersedes the previous [episub](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/episub.md) proposal. + +The proposed extensions are backwards-compatible and aim to enhance the efficiency (minimize amplification/duplicates and decrease message latency) of the gossip mesh networks by dynamically adjusting the messages sent by mesh peers based on a local view of message duplication and latency. + +In more specific terms, two new control messages are introduced, `CHOKE` and `UNCHOKE`. When a Gossipsub router is receiving many duplicates on a particular mesh, it can send a `CHOKE` message to it's mesh peers that are sending duplicates slower than its fellow mesh peers. Upon receiving a `CHOKE` message, a peer is informed to no longer propagate mesh messages to the sender of the `CHOKE` message, rather lazily (in every heartbeat) send it's gossip. + +A Gossipsub router may notice that it is receiving messages via gossip from it's `CHOKE`'d peers faster than it receives them via the mesh. In this case it may send an `UNCHOKE` message to the peer to inform the peer to resume propagating messages in the mesh. +The router may also notice that it is receiving messages from gossip from peers not in the mesh (fanout) faster than it receives messages in the mesh. It may then add these peers into the mesh. + +The modifications outlined above intend to optimize the Gossipsub mesh to receive minimal duplicates from peers with the lowest latency. + +# Specification + +## Protocol Id + +Nodes that support this Gossipsub extension should additionally advertise the version number `2.0.0`. Gossipsub nodes can advertise their own protocol-id prefix, by default this is `meshsub` giving the default protocol id: +- `/meshsub/2.0.0` + +## Parameters + +This section lists the configuration parameters that control the behaviour of the mesh optimisation. + + +| Parameter | Description | Reasonable Default | +| -------- | -------- | -------- | +| `D_non_choke` | The minimum number of peers in a mesh that must remain unchoked. | `D_lo` | +| `choke_heartbeat_interval` | The number of heartbeats before assessing and applying `CHOKE`/`UNCHOKE` control messages and adding `D_max_add`. | 20 | +| `D_max_add` | The maximum number of peers to add into the mesh (from fanout) if they are performing well per `choke_heartbeat_interval`. | 1 | +| `choke_duplicates_threshold` | The minimum number of duplicates as a percentage of received messages a peer must send before being eligible of being `CHOKE`'d. | 60 | +| `choke_churn` | The maximum number of peers that can be `CHOKE`'d or `UNCHOKE`'d in any `choke_heartbeat_interval`. | 2 | +|` unchoke_threshold` | Determines how aggressively we unchoke peers. The percentage of messages that we receive in the `choke_heartbeat_interval` that were received by gossip from a choked peer. | 50 | +| `fanout_addition_threshold` | How aggressively we add peers from the fanout into the mesh. The percentage of messages that we receive in the `choke_heartbeat_interval` that were received from a fanout peer. | 10 | + +## Choking + +Every `choke_heartbeat_interval` the router counts the number of valid (or not invalid) duplicate messages (note that the first message of its kind received is not a duplicate) and the time it took to receive each duplicate for each peer in each mesh that are not `CHOKE`'d. The router then filters which peers have sent duplicates over the `choke_duplicates_threshold` and sends `CHOKE` messages to at most `choke_churn` peers ordered by largest average latency. A router should ensure that at least `D_non_choke` peers remain in the mesh (and should not send `CHOKE` messages if this limit is to be violated) and should perform this check every heartbeat with the mesh maintenance. + +## UnChoking + +Every `choke_heartbeat_interval` the router counts the number of received valid messages obtained via `IWANT` (and hence gossip) from a `CHOKE`'d peer. If the percentage of received valid messages is greater then `unchoke_threshold` we send an `UNCHOKE` to randomly selected peers up to the `choke_churn` limit. + +## Fanout Addition + +Every `choke_heartbeat_interval` the router counts the number of received valid messages obtained via `IWANT` (and hence gossip) from `fanout` peers. If the percentage of received messages is greater than `fanout_addition_threshold` a random selection of these peers up to `D_max_add` are added to the mesh (provided the mesh bounds remain valid, i.e `D_high`). + +## Handling Gossipsub Scoring For Choked Peers + +TODO + +## Protobuf Extension + +The protobuf messages are identical to those specified in the [gossipsub v1.0.0 specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md) with the following control message modifications: + +```protobuf +message RPC { + // ... see definition in the gossipsub specification +} + +message ControlMessage { + repeated ControlIHave ihave = 1; + repeated ControlIWant iwant = 2; + repeated ControlGraft graft = 3; + repeated ControlPrune prune = 4; + repeated ControlChoke choke = 5; + repeated ControlUnChoke unchoke = 6; +} + +message ControlChoke { + optional string topicID = 1; +} + +message ControlUnChoke { + optional string topicID = 1; +} +``` From b19bcb8c2ee1eea43fb27e1e99509e33ec6aa239 Mon Sep 17 00:00:00 2001 From: Age Manning Date: Fri, 3 Jun 2022 15:25:26 +1000 Subject: [PATCH 2/4] Rename to v1.2 --- pubsub/gossipsub/{gossipsub-v2.0.md => gossipsub-v1.2.md} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename pubsub/gossipsub/{gossipsub-v2.0.md => gossipsub-v1.2.md} (98%) diff --git a/pubsub/gossipsub/gossipsub-v2.0.md b/pubsub/gossipsub/gossipsub-v1.2.md similarity index 98% rename from pubsub/gossipsub/gossipsub-v2.0.md rename to pubsub/gossipsub/gossipsub-v1.2.md index 12136b872..ddb7345c2 100644 --- a/pubsub/gossipsub/gossipsub-v2.0.md +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -18,8 +18,8 @@ The modifications outlined above intend to optimize the Gossipsub mesh to receiv ## Protocol Id -Nodes that support this Gossipsub extension should additionally advertise the version number `2.0.0`. Gossipsub nodes can advertise their own protocol-id prefix, by default this is `meshsub` giving the default protocol id: -- `/meshsub/2.0.0` +Nodes that support this Gossipsub extension should additionally advertise the version number `1.2.0`. Gossipsub nodes can advertise their own protocol-id prefix, by default this is `meshsub` giving the default protocol id: +- `/meshsub/1.2.0` ## Parameters From 847c56b6b0a8e897b1916f3c795d7a234cf3954d Mon Sep 17 00:00:00 2001 From: Age Manning Date: Wed, 9 Nov 2022 13:21:50 +1100 Subject: [PATCH 3/4] Spec improvement --- pubsub/gossipsub/gossipsub-v1.2.md | 170 ++++++++++++++++++++++++----- 1 file changed, 145 insertions(+), 25 deletions(-) diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md index ddb7345c2..03872938d 100644 --- a/pubsub/gossipsub/gossipsub-v1.2.md +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -3,22 +3,42 @@ # Overview -This document aims to provide a minimal extension to the [gossipsub v1.1](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md) protocol, that supersedes the previous [episub](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/episub.md) proposal. - -The proposed extensions are backwards-compatible and aim to enhance the efficiency (minimize amplification/duplicates and decrease message latency) of the gossip mesh networks by dynamically adjusting the messages sent by mesh peers based on a local view of message duplication and latency. - -In more specific terms, two new control messages are introduced, `CHOKE` and `UNCHOKE`. When a Gossipsub router is receiving many duplicates on a particular mesh, it can send a `CHOKE` message to it's mesh peers that are sending duplicates slower than its fellow mesh peers. Upon receiving a `CHOKE` message, a peer is informed to no longer propagate mesh messages to the sender of the `CHOKE` message, rather lazily (in every heartbeat) send it's gossip. - -A Gossipsub router may notice that it is receiving messages via gossip from it's `CHOKE`'d peers faster than it receives them via the mesh. In this case it may send an `UNCHOKE` message to the peer to inform the peer to resume propagating messages in the mesh. -The router may also notice that it is receiving messages from gossip from peers not in the mesh (fanout) faster than it receives messages in the mesh. It may then add these peers into the mesh. - -The modifications outlined above intend to optimize the Gossipsub mesh to receive minimal duplicates from peers with the lowest latency. +This document aims to provide a minimal extension to the [gossipsub +v1.1](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.1.md) +protocol, that supersedes the previous +[episub](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/episub.md) +proposal. + +The proposed extensions are backwards-compatible and aim to enhance the +efficiency (minimize amplification/duplicates and decrease message latency) of +the gossip mesh networks by dynamically adjusting the messages sent by mesh +peers based on a local view of message duplication and latency. + +In more specific terms, two new control messages are introduced, `CHOKE` and +`UNCHOKE`. When a Gossipsub router is receiving many duplicates on a particular +mesh, it can send a `CHOKE` message to it's mesh peers that are sending +duplicates slower than its fellow mesh peers. Upon receiving a `CHOKE` message, +a peer is informed to no longer propagate mesh messages to the sender of the +`CHOKE` message, rather lazily (in every heartbeat) send it's gossip. + +A Gossipsub router may notice that it is receiving messages via gossip from +it's `CHOKE`'d peers faster than it receives them via the mesh. In this case it +may send an `UNCHOKE` message to the peer to inform the peer to resume +propagating messages in the mesh. The router may also notice that it is +receiving messages from gossip from peers not in the mesh faster than it +receives messages from it's mesh peers. It may then add these peers into the +mesh. + +The modifications outlined above intend to optimize the Gossipsub mesh to +receive minimal duplicates from peers with the lowest latency. # Specification ## Protocol Id -Nodes that support this Gossipsub extension should additionally advertise the version number `1.2.0`. Gossipsub nodes can advertise their own protocol-id prefix, by default this is `meshsub` giving the default protocol id: +Nodes that support this Gossipsub extension should additionally advertise the +version number `1.2.0`. Gossipsub nodes can advertise their own protocol-id +prefix, by default this is `meshsub` giving the default protocol id: - `/meshsub/1.2.0` ## Parameters @@ -29,32 +49,95 @@ This section lists the configuration parameters that control the behaviour of th | Parameter | Description | Reasonable Default | | -------- | -------- | -------- | | `D_non_choke` | The minimum number of peers in a mesh that must remain unchoked. | `D_lo` | -| `choke_heartbeat_interval` | The number of heartbeats before assessing and applying `CHOKE`/`UNCHOKE` control messages and adding `D_max_add`. | 20 | -| `D_max_add` | The maximum number of peers to add into the mesh (from fanout) if they are performing well per `choke_heartbeat_interval`. | 1 | -| `choke_duplicates_threshold` | The minimum number of duplicates as a percentage of received messages a peer must send before being eligible of being `CHOKE`'d. | 60 | +| `choke_heartbeat_interval` | The number of heartbeats before assessing and applying `CHOKE`/`UNCHOKE` control messages and adding peers to the mesh. | 20 | | `choke_churn` | The maximum number of peers that can be `CHOKE`'d or `UNCHOKE`'d in any `choke_heartbeat_interval`. | 2 | -|` unchoke_threshold` | Determines how aggressively we unchoke peers. The percentage of messages that we receive in the `choke_heartbeat_interval` that were received by gossip from a choked peer. | 50 | -| `fanout_addition_threshold` | How aggressively we add peers from the fanout into the mesh. The percentage of messages that we receive in the `choke_heartbeat_interval` that were received from a fanout peer. | 10 | +|` unchoke_churn` | Determines how aggressively we unchoke peers. The number of peers per `choke_heartbeat_interval` that can be unchoked on an individual mesh. | 2 | +| `mesh_addition_churn` | How aggressively we add peers from into the mesh. The number of peers per `choke_heartbeat_interval` that can be added to an individual mesh. | 1 | + +## Implementation Notes + +The actual strategy for choking/unchoking peers is left to each implementation +and potentially application user. Although the strategies for choking/unchoking +can be generic, a few useful strategies are listed in the appendix for implementers. + +## The CHOKE Message + +Every `choke_heartbeat_interval` the router applies its choking strategy to a +set of collected metrics of recent messages, in order to decide if any of its +mesh peers should be choked. The router should send no more +than `choke_churn` `CHOKE` messages to peers per mesh topic. The router should +also ensure that `D_non_choke` peers remain unchoked in each mesh topic. + +Upon receiving a `CHOKE` message, the router MUST no longer forward messages to +the peer that sent the `CHOKE` message, while it is still in the mesh. Instead +it MUST always send an IHAVE message (provided it does not hit the IHAVE +message limit) in the next gossipsub heartbeat to the peer. + +A peer MUST NOT send a `CHOKE` message to another peer that is not currently +grafted into it's mesh. + +A peer MUST NOT send a `CHOKE` message to another peer that is already choked +on a given mesh topic. + +#### Pruning + +If a mesh peer sends a `PRUNE`, the local router should consider itself also +unchoked by this peer. If that peer was choked by the local router, as it is no +longer in the mesh, it should also be considered unchoked. + +Therefore, when pruning a choked peer from the mesh, an `UNCHOKE` message is +not required to be sent. + +#### Publishing + +Messages that are published to mesh peers MUST only be published to non-choked +peers. If flood-publishing, messages can be sent to non-mesh peers, which are +unchoked by definition. + +## The UNCHOKE Message -## Choking +Every `choke_heartbeat_interval` the router applies its unchoking strategy to a +set of collected metrics of recent messages, in order to decide whether to +unchoke any of it's choked peers. The router should send no more +than `unchoke_churn` `UNCHOKE` messages to peers per mesh topic. -Every `choke_heartbeat_interval` the router counts the number of valid (or not invalid) duplicate messages (note that the first message of its kind received is not a duplicate) and the time it took to receive each duplicate for each peer in each mesh that are not `CHOKE`'d. The router then filters which peers have sent duplicates over the `choke_duplicates_threshold` and sends `CHOKE` messages to at most `choke_churn` peers ordered by largest average latency. A router should ensure that at least `D_non_choke` peers remain in the mesh (and should not send `CHOKE` messages if this limit is to be violated) and should perform this check every heartbeat with the mesh maintenance. +Upon receiving an `UNCHOKE` message, the router MUST resume forwarding messages to +the peer that sent the `UNCHOKE` message and resume the normal lazy stochastic +gossiping operation in each heartbeat. -## UnChoking +A peer MUST NOT send an `UNCHOKE` message to any peer that is not currently +grafted into it's mesh. -Every `choke_heartbeat_interval` the router counts the number of received valid messages obtained via `IWANT` (and hence gossip) from a `CHOKE`'d peer. If the percentage of received valid messages is greater then `unchoke_threshold` we send an `UNCHOKE` to randomly selected peers up to the `choke_churn` limit. +A peer MUST NOT send an `UNCHOKE` message to a peer that is already unchoked on +a given mesh topic. -## Fanout Addition +## Mesh Addition -Every `choke_heartbeat_interval` the router counts the number of received valid messages obtained via `IWANT` (and hence gossip) from `fanout` peers. If the percentage of received messages is greater than `fanout_addition_threshold` a random selection of these peers up to `D_max_add` are added to the mesh (provided the mesh bounds remain valid, i.e `D_high`). +Every `choke_heartbeat_interval` the local router may use a strategy to decide +if it wishes to add peers into its meshes. Peers may be added to fill up to +`mesh_n_high` but should be limited to at most `mesh_addition_churn` per +`choke_heartbeat_interval`. -## Handling Gossipsub Scoring For Choked Peers -TODO +## Scoring for Episub + +Peers have an incentive to be choked by their mesh neighbours. Being choked +means less bandwidth the node is required to send to support the mesh network. +Malicious nodes may then intentionally attempt to game various choking +strategies in order to get choked by the router. + +Choked peers are inherently less valuable mesh peers than unchoked peers. As +such, a slight scoring penalty should be added to each choked-peer which grows +the longer they are choked. This will disfavour them when a mesh gets pruned. +As Episub introduces mesh additions, peer pruning should be more common. + +The exact scoring penalty is currently left as a TODO. ## Protobuf Extension -The protobuf messages are identical to those specified in the [gossipsub v1.0.0 specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md) with the following control message modifications: +The protobuf messages are identical to those specified in the [gossipsub v1.0.0 +specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md) +with the following control message modifications: ```protobuf message RPC { @@ -78,3 +161,40 @@ message ControlUnChoke { optional string topicID = 1; } ``` + +# Appendix + +### Optional Choking Strategies + +#### Latency Cutoff + +A mesh peer can get choked if it sends duplicates that arrive beyond a cut-off +latency. A threshold may be added such that if more than this duplicate +threshold (as a percentage) is sent over the latency threshold the peer is +eligible to be choked. + +#### Percentile Latency + +Duplicate messages can be collected over a topic and ordered by the latency +received. Peers can be choked if they send over a threshold amount of +duplicates that lie in a specific percentile. + +#### Order of Arrival + +Messages can be grouped based on the order that they arrive. If a peers average +order of messages is greater than a specified number, that peer is eligible to +be choked. + +### Optional UnChoking Strategies + +#### IHAVE Message Percentage + +If a choked peer has a sent an IHAVE message prior to mesh message for more +than a specified percent of the total mesh messages received, that peer is +eligible to be unchoked. + +### Optional Mesh Addition Strategies + +Similar to the unchoking strategy mentioned above, a router may wish to add +peers that are frequently and consistently sending IHAVE messages prior to +receiving the referenced message on the mesh. From c03418cd808c789ad4973362b32e065e10ff1cc9 Mon Sep 17 00:00:00 2001 From: Age Manning Date: Thu, 10 Nov 2022 09:46:10 +1100 Subject: [PATCH 4/4] Reviewers comments --- pubsub/gossipsub/gossipsub-v1.2.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/pubsub/gossipsub/gossipsub-v1.2.md b/pubsub/gossipsub/gossipsub-v1.2.md index 03872938d..3e0325da0 100644 --- a/pubsub/gossipsub/gossipsub-v1.2.md +++ b/pubsub/gossipsub/gossipsub-v1.2.md @@ -70,8 +70,9 @@ also ensure that `D_non_choke` peers remain unchoked in each mesh topic. Upon receiving a `CHOKE` message, the router MUST no longer forward messages to the peer that sent the `CHOKE` message, while it is still in the mesh. Instead -it MUST always send an IHAVE message (provided it does not hit the IHAVE -message limit) in the next gossipsub heartbeat to the peer. +it MUST always send an IHAVE message (provided there are messages to send and +it does not hit the IHAVE message limit) in the next gossipsub heartbeat to the +peer. A peer MUST NOT send a `CHOKE` message to another peer that is not currently grafted into it's mesh.