gossipsub v1.1 specification #254

vyzo · 2020-03-12T19:39:55Z

Specification for the changes in gossipsub v1.1.
Implementation:

…ossipsub-v1.0

pubsub/gossipsub/gossipsub-v1.0-old.md

Co-Authored-By: David Dias <[email protected]>

schomatis · 2020-03-16T05:50:14Z

This is a very detailed and well-written document that makes reviewing the implementation changes much easier, thanks @vyzo!

daviddias

There is a lot of great work here!! Really enjoyed reading through.

My review isn't fully done yet. Pushing more in just a bit

pubsub/gossipsub/README.md

pubsub/gossipsub/gossipsub-v1.0-old.md

pubsub/gossipsub/gossipsub-v1.1.md

daviddias · 2020-03-16T08:02:37Z

Note for the reviewers

There are are 4 documents in this spec:

gossipsub-v1.0.md - The spec of the v1.0 of the gossipsub protocol. This is the doc that implements have to start with
gossippsub-v1.0-old.md - This was the original spec. It packs a friendly language that is easy to parse.
gossipsub-v1.1.md - This is the document you should be reviewing!
episub.md: a research note on a protocol building on top of gossipsub to implement epidemic broadcast trees. You do not need to review this one

pubsub/gossipsub/gossipsub-v1.1.md

daviddias · 2020-03-16T08:17:53Z

pubsub/gossipsub/gossipsub-v1.1.md

+
+```
+
+### Flood Publishing


@vyzo this extension reads more as a consideration than a specification.

When should a Peer decide to use mesh/fanout or flood? Is there a randomizer for the strategy used?

It's an application option. The recommendation is that it's used, but the application may turn it off if running in a controlled environment.

daviddias · 2020-03-16T08:26:29Z

pubsub/gossipsub/gossipsub-v1.1.md

+#### Topic Parameter Calculation and Decay
+
+The topic parameters are implemented using counters maintained internally by the router
+whenever an event of interest occurs. The counters _decay_ periodically so that their values are


What is "periodically" and by how much? Is this a parameter? What are the defaults?

It is an application parameter. 1-10s is a good value.

I will add something about it in the guidelines.

daviddias · 2020-03-16T08:27:07Z

pubsub/gossipsub/gossipsub-v1.1.md

+
+#### Guidelines for Tuning the Scoring Function
+
+TBD


@vyzo any ETA?

once we've made some progress with the testground tests.

Co-Authored-By: David Dias <[email protected]>

yiannisbot

This is very nicely written and well thought-through! Some comments, mainly as clarifications. Looking forward to some tests and results!

pubsub/gossipsub/gossipsub-v1.1.md

yiannisbot · 2020-03-16T08:23:02Z

pubsub/gossipsub/gossipsub-v1.1.md

+In order to facilitate transion to the usage of signed peer records within the libp2p ecosystem,
+the emitting peer is allowed to omit the signed peer record if it doesn't have one.
+In this case, the pruned peer will have to utilize an external service to discover addresses for
+the peer, eg the DHT.


I think the last sentence means to say: ".. to discover addresses for other peers it can connect to, eg the DHT.", correct?

sure, isn't that clear?

It is, but I got confused by the "for the peer" bit. I would rephrase to "for other peers it can connect to". But it's a really minor issue.

pubsub/gossipsub/gossipsub-v1.1.md

pubsub/gossipsub/gossipsub-v1.0-old.md

vasco-santos · 2020-03-16T16:48:21Z

pubsub/gossipsub/README.md

+- [gossipsub-v1.0](gossipsub-v1.0.md): v1.0 of the gossipsub protocol. This is a revised specification, to use a more normative language. 
+  - The original v1.0 specification is [here](gossippsub-v1.0-old.md).
+- [gossipsub-v1.1](gossipsub-v1.1.md): v1.1 of the gossipsub protocol.
+- [(not in use) episub](episub.md): a research note on a protocol building on top of gossipsub to implement [epidemic broadcast trees](https://www.gsd.inesc-id.pt/~ler/reports/srds07.pdf).

 ## Implementation status


Should we add to the implementation status the version of gossipsub currently released? Or should it be a responsibility of the implementation to make it clear?

Also, gossipsub-js is currently released and compliant with v1.0, so the current "work in progress; check branches and PRs" notice is outdated. Should I updated it in a different PR?

you can make a PR on top of this PR and we can merge it.

Yeah, having a table to say which ones implement what would be ideal

pubsub/gossipsub/gossipsub-v1.1.md

Co-Authored-By: Vasco Santos <[email protected]>

petar · 2020-03-17T04:32:59Z

pubsub/gossipsub/gossipsub-v1.1.md

+
+In gossipsub v1.0 a freshly published message is propagated through the mesh or the fanout map
+if the publisher is not subscribed to the topic. In gossipsub v1.1 publishing is (optionally)
+done by publishing the message to all connected peers with a score above a publish threshold


I was mistakenly thinking that scores were meant to be communicated to other peers. I think I understand now: Each node keeps track of the scores of its "connected" nodes. Then it uses this information privately to sub-select from the scored nodes which ones to send flood messages to. (And same for adaptive gossip, below).

"Sybil" nodes are then definitionally the nodes that have low scores (i.e. low contribution within some historical window of time). And the algorithm ensures routing around them.

Yes, exactly.

Then it uses this information privately to sub-select from the scored nodes which ones to send flood messages to. (And same for adaptive gossip, below).

Flood messages are sent everywhere. The score is mainly used by nodes to decide which nodes to keep in its mesh and which ones to PRUNE.

Having a "clean" mesh of honest nodes is the most important to defend against pretty much any attack.

petar · 2020-03-17T04:39:08Z

pubsub/gossipsub/gossipsub-v1.1.md

+a `0.578125` probability.
+
+This behaviour is prescribed to counter sybil attacks and ensures that a message from a honest
+node propagates in the network with high probability.


This statement does not sound 100% accurate. Sybil attacks are not entirely "countered" with high probability. For instance, if 80% of the nodes are sybils and they invest enough time/work for the scoring function to reach its decay threshold, then they will be everywhere in the network, and then they can disconnect it. I think you mean something along the lines of: Sybil attacks whose resources are below a certain limit amount (of work, or something else) will be countered with high probability. And, of course, probably outside of the scope of a spec one would want to compute the exact amount of work that suffices to take down a network of a given size.

For instance, if 80% of the nodes are sybils and they invest enough time/work for the scoring function to reach its decay threshold, then they will be everywhere in the network, and then they can disconnect it.

This measure of increased gossip dissemination is not exactly about countering the attack itself. It's about making sure that even in this case, some honest nodes have good chances of receiving messages. So, in case of 80% of Sybils (that have not gone below the gossipThreshold), there is still ~(0.578*0.20)=11% chance that they will receive the gossip.

Those that have gone below the gossipThreshold are not "entitled" to received gossip messages and are, therefore, excluded from this group - hence, the above probability should go further up.

Not sure where you are getting this 11% from. Even with 90% sybils, a little over than half the honest nodes will get the message via gossip (even if the mesh breaks down completely by being taken over by sybils somehow).
This half will propagate to the other half in a few hops at most, with high probability.

But sybils will be among the nodes we're gossiping with too (with the same probability), no? In this case, only the percentage of honest nodes times 0.57 will receive the messages. That's how I came up with the 11%.

petar

I think gossipsub-v1.0 and v1.1 are clearly written and the mesh design is understandable and natural.
I had one misunderstanding about the scoring function, which I believe is cleared now (I've left a couple of comments.). LGTM

chore: gossipsub js status update

vyzo added 10 commits March 12, 2020 14:31

move gossipsub-wip into the gossipsub directory

bcf0ae1

move original specification to gossipsub-v1.0-old, gossipsub-wip to g…

96eb7a5

…ossipsub-v1.0

add a README that points to the specifications.

4a4f0e0

add editor to gossipsub-v1.0 spec

8e4aa08

gossipsub-v.1.1 stub

ee4090a

overview and px

82e4c54

flood publishing

8cb2f40

adaptive gossip dissemination

d798573

peer scoring preliminaries

93d58e1

peer scoring function

b65be4e

daviddias requested review from daviddias, dirkmc, Kubuxu, petar, raulk, whyrusleeping, yiannisbot and yusefnapora March 13, 2020 16:23

daviddias reviewed Mar 13, 2020

View reviewed changes

pubsub/gossipsub/gossipsub-v1.0-old.md Outdated Show resolved Hide resolved

daviddias reviewed Mar 13, 2020

View reviewed changes

pubsub/gossipsub/gossipsub-v1.0-old.md Show resolved Hide resolved

daviddias reviewed Mar 13, 2020

View reviewed changes

pubsub/gossipsub/gossipsub-v1.0-old.md Outdated Show resolved Hide resolved

vyzo and others added 5 commits March 13, 2020 18:30

Update pubsub/gossipsub/gossipsub-v1.0-old.md interest group

064250f

Co-Authored-By: David Dias <[email protected]>

Update pubsub/gossipsub/gossipsub-v1.0-old.md interest group

c99c7bf

Co-Authored-By: David Dias <[email protected]>

Update pubsub/gossipsub/gossipsub-v1.0-old.md

c3f172d

Co-Authored-By: David Dias <[email protected]>

score parameter calculation

3c0d7df

spam protection measures

87ca053

daviddias approved these changes Mar 16, 2020

View reviewed changes

daviddias force-pushed the rfc/gossipsub-v1.1 branch from f1fd0bc to 43b506e Compare March 16, 2020 07:59

docs: gossipsub specs review

a38055f

daviddias force-pushed the rfc/gossipsub-v1.1 branch from 43b506e to a38055f Compare March 16, 2020 08:00

daviddias reviewed Mar 16, 2020

View reviewed changes

pubsub/gossipsub/gossipsub-v1.1.md Outdated Show resolved Hide resolved

daviddias reviewed Mar 16, 2020

View reviewed changes

pubsub/gossipsub/gossipsub-v1.1.md Show resolved Hide resolved

daviddias reviewed Mar 16, 2020

View reviewed changes

Update pubsub/gossipsub/gossipsub-v1.1.md

e8a163f

Co-Authored-By: David Dias <[email protected]>

yiannisbot reviewed Mar 16, 2020

View reviewed changes

some clarifications for PX

e7a4d32

vasco-santos reviewed Mar 16, 2020

View reviewed changes

fix typo

c681902

Co-Authored-By: Vasco Santos <[email protected]>

petar reviewed Mar 17, 2020

View reviewed changes

petar approved these changes Mar 17, 2020

View reviewed changes

vasco-santos and others added 2 commits March 17, 2020 11:18

chore: gossipsub-js status update

1ebd52b

Merge pull request #256 from libp2p/chore/gossipsub-js-status-update

7f5e049

chore: gossipsub js status update

vasco-santos approved these changes Mar 17, 2020

View reviewed changes

fix typo, toc, rename attack section

1e029d8

yiannisbot approved these changes Mar 17, 2020

View reviewed changes

daviddias changed the title ~~[WIP] gossipsub v1.1~~ gossipsub v1.1 specification Mar 20, 2020

daviddias added 2 commits March 20, 2020 07:34

tidy up

3bd25cb

add jvm and py

2fcfd80

daviddias merged commit 2ed2277 into master Mar 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gossipsub v1.1 specification #254

gossipsub v1.1 specification #254

vyzo commented Mar 12, 2020

schomatis commented Mar 16, 2020

daviddias left a comment

daviddias commented Mar 16, 2020 •

edited

Loading

daviddias Mar 16, 2020

vyzo Mar 16, 2020 •

edited

Loading

daviddias Mar 16, 2020

vyzo Mar 16, 2020

vyzo Mar 16, 2020

daviddias Mar 16, 2020

vyzo Mar 16, 2020

yiannisbot left a comment

yiannisbot Mar 16, 2020

vyzo Mar 16, 2020

yiannisbot Mar 16, 2020

vasco-santos Mar 16, 2020

vasco-santos Mar 16, 2020

vyzo Mar 16, 2020

daviddias Mar 20, 2020

petar Mar 17, 2020

yiannisbot Mar 17, 2020

petar Mar 17, 2020

yiannisbot Mar 17, 2020

vyzo Mar 17, 2020

yiannisbot Mar 19, 2020

petar left a comment


		```

		### Flood Publishing

gossipsub v1.1 specification #254

gossipsub v1.1 specification #254

Conversation

vyzo commented Mar 12, 2020

schomatis commented Mar 16, 2020

daviddias left a comment

Choose a reason for hiding this comment

daviddias commented Mar 16, 2020 • edited Loading

Note for the reviewers

Choose a reason for hiding this comment

vyzo Mar 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yiannisbot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petar left a comment

Choose a reason for hiding this comment

daviddias commented Mar 16, 2020 •

edited

Loading

vyzo Mar 16, 2020 •

edited

Loading