Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation of gossipsub v1.1 - Hardening #263

Merged
merged 94 commits into from
Apr 18, 2020
Merged

Implementation of gossipsub v1.1 - Hardening #263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
8146736
gomod: use go-libp2p-core@peer-records and go-libp2p-peerstore@certif…
vyzo Nov 21, 2019
04f2211
track changes to peer records in -core
yusefnapora Jan 22, 2020
4ddbfdc
update PR branch dependencies
yusefnapora Jan 23, 2020
d75f82d
gomod: use go-libp2p-core@peer-records and go-libp2p-peerstore@certif…
vyzo Nov 21, 2019
cdf701c
hook the tracer for peer scoring
vyzo Feb 28, 2020
2fbd21b
peer score scaffolding
vyzo Mar 1, 2020
b3f4b96
hook peer score into control message handlers
vyzo Mar 2, 2020
31b499b
filter peers with negative score in initial mesh on join
vyzo Mar 2, 2020
5796368
heartbeat doPX is a more complex decision
vyzo Mar 2, 2020
9b0494c
hook score into heartbeat maintenance
vyzo Mar 2, 2020
7e19b06
remove peers with negative score from fanout
vyzo Mar 2, 2020
8711fae
keep D peer when pruning for oversubscription
vyzo Mar 3, 2020
8ca01cb
implement flood publishing option
vyzo Mar 3, 2020
516d1fe
correctly implement flood publishing
vyzo Mar 3, 2020
c58601d
increase D_score to 4
vyzo Mar 3, 2020
54059a9
backoff when we prune too, refuse to reGRAFT backed off peers
vyzo Mar 3, 2020
ce10a1d
router.Publish doesn't need the source, it takes the whole message now
vyzo Mar 4, 2020
0df8efd
allow gossipsub to graylist peers based on score
vyzo Mar 4, 2020
06dad9b
select good peers for PX, tune the px related limits
vyzo Mar 4, 2020
063c108
adaptive gossip dissemination
vyzo Mar 5, 2020
74ceaed
track peer (re)transmissions in message cache
vyzo Mar 5, 2020
88ca06b
limit message retransmissions through IWANT requests
vyzo Mar 5, 2020
79468b7
peer score parameters
vyzo Mar 7, 2020
e501ee2
explicit Start for peer scoring to inject the router
vyzo Mar 7, 2020
7caeed5
peer score calculation
vyzo Mar 7, 2020
d66c04f
peer score refresh: counter decay and mesh activation
vyzo Mar 7, 2020
b7a04c2
fix peer IP tracking structure
vyzo Mar 7, 2020
62a3084
add nil check in Score function
vyzo Mar 7, 2020
f5f8f6c
respec decay to zero parameter in counter decay
vyzo Mar 8, 2020
12b667e
respect the time in mesh cap
vyzo Mar 8, 2020
05baab3
message delivery tracking
vyzo Mar 8, 2020
f1df2a8
minimize hash table accesses
vyzo Mar 8, 2020
30fdeae
mesh tracking
vyzo Mar 8, 2020
cb68449
peer tracking
vyzo Mar 8, 2020
c0d94c2
track message delivery time within the validation pipeline
vyzo Mar 8, 2020
6d28a50
comment cosmetics
vyzo Mar 8, 2020
66c30c1
simplify duplicate mesh delivery tracking
vyzo Mar 8, 2020
ef59ed0
more logical code ordering
vyzo Mar 8, 2020
990d789
clean up the logic of mesh delivery window
vyzo Mar 8, 2020
54a50b7
sticky mesh failure penalty parameter
vyzo Mar 8, 2020
48cd691
finetune sticky mesh failure penalty parameter
vyzo Mar 8, 2020
bbc530e
double check delivery to aboid duplicate counts
vyzo Mar 9, 2020
863ee6b
implement message delivery struct details
vyzo Mar 9, 2020
0e6658c
implement getIPs and refreshIPs
vyzo Mar 9, 2020
79b0cc8
implement periodic maintenance
vyzo Mar 9, 2020
17be73d
peer score parameter validation
vyzo Mar 9, 2020
31729dc
fix bug in WithEventTracer
vyzo Mar 11, 2020
f37aefe
distinguish throttled validation because of a full worker queue.
vyzo Mar 16, 2020
e211072
short-circuit score computation in peer selection filters
vyzo Mar 16, 2020
272b374
move score starting nil check inside the score implementation
vyzo Mar 16, 2020
3fb634b
validate the threshold values
vyzo Mar 16, 2020
49610c1
add option to periodically inspect peer scores for debugging purposes
vyzo Mar 17, 2020
db8c95e
shuffle IWANT list
vyzo Mar 22, 2020
aa2398b
refresh backoff when pruning during the backoff period
vyzo Mar 22, 2020
6b296fd
add backoff when pruning because of negative score
vyzo Mar 23, 2020
5df605b
check exact expiration date of backoff when pruning because of backoff
vyzo Mar 23, 2020
07618dc
track heartbeat ticks, only clear backoff every 15.
vyzo Mar 23, 2020
9aaa1ae
fix rebase artifacts
vyzo Mar 24, 2020
2c3b085
gomod tidy
vyzo Mar 24, 2020
0fc539c
move score parameter types to their own file
vyzo Mar 27, 2020
713818a
make PX optional and disabled by default, gate by acceptPXThreshold
vyzo Mar 27, 2020
1f33460
use a struct for aggregating score thresholds
vyzo Mar 27, 2020
c0b3c71
reraise GossipSubPrunePeers to 16
vyzo Mar 27, 2020
6e046e1
rename score debug option, address some comments around inspection
vyzo Mar 27, 2020
a19cc92
wrap topic param validation error
vyzo Mar 27, 2020
f3aaa16
godoc nits for parameter weights.
vyzo Mar 27, 2020
b148380
topic score cap
vyzo Mar 27, 2020
f804e71
only retain negative scores
vyzo Mar 27, 2020
e135d52
increase connection manager weight for mesh peers
vyzo Mar 27, 2020
9ae9c41
don't forward messages with source ourselves but not published by our…
vyzo Mar 27, 2020
fdef175
gomod: use go-multiaddr-net v0.1.3
vyzo Mar 27, 2020
dc65a80
use manet.ToIP instead of hand-rolling multiaddr to IP conversion
vyzo Mar 27, 2020
1782934
add IP whitelist for colocation factor penalty
vyzo Mar 27, 2020
a9a47ca
use named constants for tracing message rejection reasons
vyzo Mar 27, 2020
f4df765
rename delivery_ constants to satisfy the camelcase worshipers and ni…
vyzo Mar 27, 2020
879b768
move self-publish check to pubsub
vyzo Mar 27, 2020
f8a84fc
better handling of message originating from self but forwarded by ano…
vyzo Mar 28, 2020
02235f2
correctly attribute invalid message deliveries due to self origin rej…
vyzo Mar 28, 2020
bc85b64
update REAMDE
vyzo Mar 31, 2020
1951023
support direct peering agreements
vyzo Mar 31, 2020
a756f34
comments and nits.
raulk Apr 1, 2020
65cd590
eagerly send gossip at the end heartbeat
vyzo Apr 3, 2020
ce79951
use a configurable variable for direct peer reconnect ticks
vyzo Apr 10, 2020
44ae7ac
opportunistic grafting
vyzo Apr 10, 2020
6f5f9aa
fix typo
vyzo Apr 10, 2020
8947160
fix typo
vyzo Apr 10, 2020
69d393f
configurable number of peers for opportunistic grafting
vyzo Apr 10, 2020
8e41071
disable flaky and irrelevant tests
vyzo Apr 11, 2020
d34bd94
fix bug in peerScore.setIPs
vyzo Apr 14, 2020
e640bed
gomod: update go-multiaddr-net
vyzo Apr 14, 2020
ed42cfd
fix issue with unspecific IPv6 addrs in conn IP tracking
vyzo Apr 14, 2020
9b54f0c
more rigorously defeat unspecified address gremlins
vyzo Apr 14, 2020
c7fb1e6
fix IPv4 -vs- IPv6 address handling in peer address tracking
vyzo Apr 14, 2020
e92f1eb
gomod tidy
vyzo Apr 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ This repo contains the canonical pubsub implementation for libp2p. We currently

[@vyzo](https://github.com/vyzo/)

> This repo follows the [Repo Lead Maintainer Protocol](https://github.com/ipfs/team-mgmt/blob/master/LEAD_MAINTAINER_PROTOCOL.md)
> This repo follows the [Repo Lead Maintainer Protocol](https://github.com/ipfs/team-mgmt/blob/master/LEAD_MAINTAINER_PROTOCOL.md)

## Table of Contents

Expand Down Expand Up @@ -88,10 +88,12 @@ See the [libp2p specs](https://github.com/libp2p/specs/tree/master/pubsub) for h
├── validation.go
# Floodsub router
├── floodsub.go
# Randomsub router
# Randomsub router
├── randomsub.go
# Gossipsub router
# Gossipsub router
├── gossipsub.go
├── score.go
├── score_params.go
└── mcache.go
```

Expand Down
1 change: 1 addition & 0 deletions discovery_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,7 @@ func TestSimpleDiscovery(t *testing.T) {
}

func TestGossipSubDiscoveryAfterBootstrap(t *testing.T) {
t.Skip("flaky test disabled")
ctx, cancel := context.WithCancel(context.Background())
defer cancel()

Expand Down
12 changes: 8 additions & 4 deletions floodsub.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@ package pubsub
import (
"context"

pb "github.com/libp2p/go-libp2p-pubsub/pb"

"github.com/libp2p/go-libp2p-core/host"
"github.com/libp2p/go-libp2p-core/peer"
"github.com/libp2p/go-libp2p-core/protocol"
Expand Down Expand Up @@ -69,9 +67,15 @@ func (fs *FloodSubRouter) EnoughPeers(topic string, suggested int) bool {
return false
}

func (fs *FloodSubRouter) AcceptFrom(peer.ID) bool {
return true
}

func (fs *FloodSubRouter) HandleRPC(rpc *RPC) {}

func (fs *FloodSubRouter) Publish(from peer.ID, msg *pb.Message) {
func (fs *FloodSubRouter) Publish(msg *Message) {
from := msg.ReceivedFrom

tosend := make(map[peer.ID]struct{})
for _, topic := range msg.GetTopicIDs() {
tmap, ok := fs.p.topics[topic]
Expand All @@ -84,7 +88,7 @@ func (fs *FloodSubRouter) Publish(from peer.ID, msg *pb.Message) {
}
}

out := rpcWithMessages(msg)
out := rpcWithMessages(msg.Message)
for pid := range tosend {
if pid == from || pid == peer.ID(msg.GetFrom()) {
continue
Expand Down
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ require (
github.com/libp2p/go-libp2p-swarm v0.2.2
github.com/libp2p/go-libp2p-transport-upgrader v0.2.0 // indirect
github.com/multiformats/go-multiaddr v0.2.1
github.com/multiformats/go-multiaddr-net v0.1.4
github.com/multiformats/go-multistream v0.1.1
github.com/whyrusleeping/timecache v0.0.0-20160911033111-cfcb2f1abfee
golang.org/x/crypto v0.0.0-20200117160349-530e935923ad // indirect
Expand Down
12 changes: 12 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ github.com/ipfs/go-ds-badger v0.0.5/go.mod h1:g5AuuCGmr7efyzQhLL8MzwqcauPojGPUaH
github.com/ipfs/go-ds-badger v0.0.5/go.mod h1:g5AuuCGmr7efyzQhLL8MzwqcauPojGPUaHzfGTzuE3s=
github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc=
github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc=
github.com/ipfs/go-ds-leveldb v0.0.1/go.mod h1:feO8V3kubwsEF22n0YRQCffeb79OOYIykR4L04tMOYc=
github.com/ipfs/go-ipfs-delay v0.0.0-20181109222059-70721b86a9a8/go.mod h1:8SP1YXK1M1kXuc4KJZINY3TQQ03J2rwBG9QfXmbRPrw=
github.com/ipfs/go-log v0.0.1 h1:9XTUN/rW64BCG1YhPK9Hoy3q8nr4gOmHHBpgFdfw6Lc=
github.com/ipfs/go-log v0.0.1/go.mod h1:kL1d2/hzSpI0thNYjiKfjanbVNU+IIGA/WnNESY9leM=
Expand Down Expand Up @@ -210,6 +211,8 @@ github.com/multiformats/go-multiaddr v0.0.2/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lg
github.com/multiformats/go-multiaddr v0.0.4/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44=
github.com/multiformats/go-multiaddr v0.1.0/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44=
github.com/multiformats/go-multiaddr v0.1.1/go.mod h1:aMKBKNEYmzmDmxfX88/vz+J5IU55txyt0p4aiWVohjo=
github.com/multiformats/go-multiaddr v0.2.0 h1:lR52sFwcTCuQb6bTfnXF6zA2XfyYvyd+5a9qECv/J90=
github.com/multiformats/go-multiaddr v0.2.0/go.mod h1:0nO36NvPpyV4QzvTLi/lafl2y95ncPj0vFwVF6k6wJ4=
github.com/multiformats/go-multiaddr v0.2.0/go.mod h1:0nO36NvPpyV4QzvTLi/lafl2y95ncPj0vFwVF6k6wJ4=
github.com/multiformats/go-multiaddr v0.2.1 h1:SgG/cw5vqyB5QQe5FPe2TqggU9WtrA9X4nZw7LlVqOI=
github.com/multiformats/go-multiaddr v0.2.1/go.mod h1:s/Apk6IyxfvMjDafnhJgJ3/46z7tZ04iMk5wP4QMGGE=
Expand All @@ -223,6 +226,8 @@ github.com/multiformats/go-multiaddr-net v0.1.0 h1:ZepO8Ezwovd+7b5XPPDhQhayk1yt0
github.com/multiformats/go-multiaddr-net v0.1.0/go.mod h1:5JNbcfBOP4dnhoZOv10JJVkJO0pCCEf8mTnipAo2UZQ=
github.com/multiformats/go-multiaddr-net v0.1.2 h1:P7zcBH9FRETdPkDrylcXVjQLQ2t1JQtNItZULWNWgeg=
github.com/multiformats/go-multiaddr-net v0.1.2/go.mod h1:QsWt3XK/3hwvNxZJp92iMQKME1qHfpYmyIjFVsSOY6Y=
github.com/multiformats/go-multiaddr-net v0.1.4 h1:g6gwydsfADqFvrHoMkS0n9Ok9CG6F7ytOH/bJDkhIOY=
github.com/multiformats/go-multiaddr-net v0.1.4/go.mod h1:ilNnaM9HbmVFqsb/qcNysjCu4PVONlrBZpHIrw/qQuA=
github.com/multiformats/go-multibase v0.0.1 h1:PN9/v21eLywrFWdFNsFKaU04kLJzuYzmrJR+ubhT9qA=
github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs=
github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U=
Expand All @@ -231,13 +236,19 @@ github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa
github.com/multiformats/go-multihash v0.0.13 h1:06x+mk/zj1FoMsgNejLpy6QTvJqlSt/BhLEy87zidlc=
github.com/multiformats/go-multihash v0.0.13/go.mod h1:VdAWLKTwram9oKAatUcLxBNUjdtcVwxObEQBtRfuyjc=
github.com/multiformats/go-multistream v0.1.0 h1:UpO6jrsjqs46mqAK3n6wKRYFhugss9ArzbyUzU+4wkQ=
github.com/multiformats/go-multistream v0.1.0 h1:UpO6jrsjqs46mqAK3n6wKRYFhugss9ArzbyUzU+4wkQ=
github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg=
github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg=
github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wSzZ5NwG2FEVAI30fiovg=
github.com/multiformats/go-multistream v0.1.1 h1:JlAdpIFhBhGRLxe9W6Om0w++Gd6KMWoFPZL/dEnm9nI=
github.com/multiformats/go-multistream v0.1.1/go.mod h1:KmHZ40hzVxiaiwlj3MEbYgK9JFk2/9UktWZAF54Du38=
github.com/multiformats/go-varint v0.0.1 h1:TR/0rdQtnNxuN2IhiB639xC3tWM4IUi7DkTBVTdGW/M=
github.com/multiformats/go-varint v0.0.1/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
github.com/multiformats/go-varint v0.0.1/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
github.com/multiformats/go-varint v0.0.2 h1:6sUvyh2YHpJCb8RZ6eYzj6iJQ4+chWYmyIHxszqlPTA=
github.com/multiformats/go-varint v0.0.2 h1:6sUvyh2YHpJCb8RZ6eYzj6iJQ4+chWYmyIHxszqlPTA=
github.com/multiformats/go-varint v0.0.2/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
github.com/multiformats/go-varint v0.0.2/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
github.com/multiformats/go-varint v0.0.5 h1:XVZwSo04Cs3j/jS0uAEPpT3JY6DzMcVLLoWOSnCxOjg=
github.com/multiformats/go-varint v0.0.5/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
Expand Down Expand Up @@ -363,6 +374,7 @@ golang.org/x/tools v0.0.0-20190311212946-11955173bddd h1:/e+gpKk9r3dJobndpTytxS2
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7 h1:9zdDQZ7Thm29KFXgAX/+yaf3eVbP7djjWp/dXAppNCc=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
Expand Down
Loading