Initial Whisper implementation #6009

rphmeier · 2017-07-06T10:25:51Z

Summary of changes:

whisper::net module contains the network handler, which maintains a pool of messages up to a given size. Messages with a higher PoW will push out those with lower.
We respect other peers' topic bloom filters, but do not issue them ourselves (TODO)
The RPC layer expects a common payload format, as well as three kinds of encryption:
1. Asymmetric: ECIES
2. Symmetric AES-GCM with multi-use key, random nonce appended to ciphertext
3. Broadcast: Symmetric AES-GCM with single use key and global nonce, with key XOR'd into topic hashes. "salted topics" prepended to ciphertext.
The payload format consists of the message body, arbitrary padding, and an optional signature.
There are RPCs for creating ephemeral identities, either asymmetric (Secp256k1) or symmetric (AES-256-GCM).
Enable Whisper with the --whisper flag, "shh" RPC API, and choose target memory (MB) with --whisper-pool-size. This could be made dynamic in the future.
Subscribe or poll filters for messages with certain topics, signatures, and encryption/broadcast mode

rphmeier · 2017-07-06T10:38:14Z

I'm OK with adding a scheme for giving out topic bloom filters to peers, but I'd like the "darkness" to be configurable.

The rough idea is to take the list of each bloom relevant to a local subscription and give each of N peers 1 / N of them xor'd together, with a few random bits set. The hard part is making sure we always have coverage of all the subscriptions we care about even in the face of a changing or unreliable peer set.

An optional darkness parameter when creating a subscription which controls how much information we leak about that subscription would be a further goal.

rphmeier · 2017-07-06T15:50:35Z

(I'll ice this to post 1.7.0)

arkpar · 2017-07-14T11:32:05Z

ethkey/src/random.rs

+
+	fn generate(self) -> Result<KeyPair, Self::Error> {
+		let (sec, publ) = SECP256K1.generate_keypair(self)
+			.expect("context always created with full capabilities; qed");


Is invalid capabilities the only possible source of failure here? Why not just pass the error as it was before the change?

it's the only source of failure, yeah. @debris and I discussed making this function infallible before, and I figured this was a good time for it.

arkpar · 2017-07-14T11:40:05Z

whisper/src/message.rs

+
+/// A topic of a message.
+#[derive(Debug, Default, Clone, Copy, Hash, PartialEq, Eq, PartialOrd, Ord)]
+pub struct Topic(pub [u8; 4]);


Why not use H32? It supports RLP serialization and bloom filters already.

The bloom algorithm is slightly different, I think. It's not too much overhead to have a separate type, and it conveys the usage better IMO

arkpar · 2017-07-14T12:12:29Z

whisper/src/message.rs

+		use rand::{Rng, SeedableRng, XorShiftRng};
+
+		let mut rng = {
+			let mut thread_rng = ::rand::thread_rng();


Does it not require a crypto-secure rng?

it's only used to find a nonce which brings the PoW below the threshold. Speed here would be the critical factor. From an outside observer's perspective it doesn't matter whether the method used to choose the nonce is high or low entropy. It could even be consecutive

arkpar · 2017-07-14T12:22:44Z

whisper/src/message.rs

+		if envelope.expiry <= envelope.ttl { return Err(Error::LivesTooLong) }
+		if envelope.ttl == 0 { return Err(Error::ZeroTTL) }
+
+		let issue_time_adjusted = Duration::from_secs(envelope.expiry - envelope.ttl - LEEWAY_SECONDS);


can this underflow?

at least the subtraction of leeway can. I'll make that saturating.

arkpar · 2017-07-14T12:30:57Z

whisper/src/net.rs

+
+	// whether a message is known or within the bounds of PoW.
+	fn may_accept(&self, message: &Message) -> bool {
+		!self.known.contains(message.hash()) && {


Comment above says that a known message should return true

yeah, should be is not known (I had it as is_known before but forgot to update when I generalized it)

arkpar · 2017-07-14T12:38:17Z

whisper/src/net.rs

+// when full, will accept messages above the minimum stored.
+struct Messages {
+	slab: ::slab::Slab<Message>,
+	sorted: Vec<SortedEntry>,


perhaps std::collections::BinaryHeap would fit better?

a binary heap will be optimized for the use case when we tend to pop off the first element. we tend to remove elements by two metrics: when they expire (~random access) and also by pushing out low PoW elements (sorted access). My intuition was that vector pops (removing low work entries) are very cheap O(n) worst case, and a vector retain periodically to prune expired messages is also relatively cheap O(n).

Doing an arbitrary number of pops from a binary heap to remove low work entries is O(nlogn), and the retainment logic for live messages would have to reconstruct the heap each time from the iterator, again O(nlogn)

arkpar · 2017-07-14T12:47:34Z

whisper/src/net.rs

+			let mut peer = peer.lock();
+
+			if !peer.can_send_messages() {
+				return Err(Error::UnexpectedMessage);


Can connection recover after this? Maybe it would be better to disconnect on any such error?

the error will lead to a punishment in the caller of this function

rphmeier added 30 commits June 26, 2017 15:35

whisper skeleton

b7708b5

basic message store

0ff28fe

rallying and message logic

5c8a374

pass host info to network protocol handlers

125389e

choose who starts rally based on node key

61acfc6

module reshuffling

7ecbb7f

mining messages

8122805

prune messages by low PoW until below size target

1161a76

Merge branch 'master' into whisper

67a51d2

associated error type for ethkey generators and OsRng generator

32575c6

beginnings of RPC

8bff980

generic message handler for whisper

fdb8cbc

reshuffle code order

435d494

standard payload encoding and decoding

4332566

basic crypto

79f6bfe

minor restructuring of net code

953ad99

implement shh_post

9d09a67

merge?

a2af2e5

Merge branch 'master' into whisper

601f6cd

implement filters

6837950

rand trait for hash types

975fb81

filter RPCs for whisper

03a839c

symmetric encryption of payload

034db4e

pub-sub

2041439

filter tests

6b8cd61

use only secure random IDs

7261b0c

attach arbitrary protocols to network

e504198

basic integration of whisper into Parity

6acf0b5

eagerly prune low PoW entries

2cf05a7

broadcast messages with salted topics

1e31280

rphmeier added the M4-core ⛓ Core client code / Rust. label Jul 6, 2017

Merge github.com:paritytech/parity into whisper

ca8f8eb

rphmeier force-pushed the whisper branch from ddbd3af to ca8f8eb Compare July 6, 2017 13:22

rphmeier mentioned this pull request Jul 6, 2017

migration to serde 1.0 #5996

Merged

rphmeier added A1-onice 🌨 Pull request is reviewed well, but should not yet be merged. and removed A0-pleasereview 🤓 Pull request needs code review. labels Jul 6, 2017

rphmeier added 3 commits July 6, 2017 18:08

fix import

921f0d3

fix leading zeros calculation

a7bb56a

Merge branch 'master' into whisper

79f03ea

rphmeier added A0-pleasereview 🤓 Pull request needs code review. and removed A1-onice 🌨 Pull request is reviewed well, but should not yet be merged. labels Jul 13, 2017

rphmeier added 2 commits July 13, 2017 17:45

Merge branch 'master' into whisper

7431461

Merge branch 'master' into whisper

9f0741f

rphmeier force-pushed the whisper branch from 3eff9f0 to 9f0741f Compare July 14, 2017 03:14

arkpar reviewed Jul 14, 2017

View reviewed changes

arkpar added A5-grumble 🔥 Pull request has minor issues that must be addressed before merging. and removed A0-pleasereview 🤓 Pull request needs code review. labels Jul 14, 2017

address minor grumbles

69b2c7c

arkpar added A8-looksgood 🦄 Pull request is reviewed well. and removed A5-grumble 🔥 Pull request has minor issues that must be addressed before merging. labels Jul 14, 2017

gavofyork merged commit 99075ad into master Jul 14, 2017

gavofyork deleted the whisper branch July 14, 2017 18:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial Whisper implementation #6009

Initial Whisper implementation #6009

rphmeier commented Jul 6, 2017 •

edited

Loading

rphmeier commented Jul 6, 2017 •

edited

Loading

rphmeier commented Jul 6, 2017

arkpar Jul 14, 2017

rphmeier Jul 14, 2017

arkpar Jul 14, 2017

rphmeier Jul 14, 2017

arkpar Jul 14, 2017

rphmeier Jul 14, 2017

arkpar Jul 14, 2017

rphmeier Jul 14, 2017

arkpar Jul 14, 2017

rphmeier Jul 14, 2017

arkpar Jul 14, 2017

rphmeier Jul 14, 2017

arkpar Jul 14, 2017

rphmeier Jul 14, 2017 •

edited

Loading

Initial Whisper implementation #6009

Initial Whisper implementation #6009

Conversation

rphmeier commented Jul 6, 2017 • edited Loading

rphmeier commented Jul 6, 2017 • edited Loading

rphmeier commented Jul 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rphmeier Jul 14, 2017 • edited Loading

Choose a reason for hiding this comment

rphmeier commented Jul 6, 2017 •

edited

Loading

rphmeier commented Jul 6, 2017 •

edited

Loading

rphmeier Jul 14, 2017 •

edited

Loading