[wip] iroh-memesync #586

dignifiedquire · 2022-12-12T18:43:42Z

Opening this mostly for visibility, but it does transfer some bytes around

…iroh` commands (#348) * docs: add full help text for `lookup`, `connect`, `get`, `p2p`, and `iroh` commands * move long descriptions into constants Co-authored-by: b5 <[email protected]>

hannahhoward

I'm sure you don't expect anyone reviewing this but it's fun to follow a long your work. Generally looks awesome.

hannahhoward · 2022-12-28T23:33:09Z

iroh-memesync/src/message.rs

+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, Hash)]
+pub struct QueryId(u64);


make it a UUID... you'll be happy you did in the future :P

some rando browsing this code

what would be the practical benefit? this only needs to be unique for the sender-receiver pair, and even if it is not, not much bad happens. uuids are considerably larger though

we had uints as graphsync ids originally.
we eventual made it a UUID, largely to remove a bunch of cruft code matching up IDs with layered protocols like go-data-transfer and filecoin retrieval. Plus I find tracking things with just a single id quite pleasant (as opposed to always appending peer ids)
yes they are larger, it's certainly a tradeoff, maybe not worth making now.
the nice thing about uuids is you can share them between layered or seperate protocols.

hannahhoward · 2022-12-28T23:38:05Z

iroh-memesync/src/message.rs

+impl Message {
+    /// Decoding from a given `BytesMut`.
+    pub fn from_bytes(bytes: BytesMut) -> Result<Self, Error> {
+        let msg = bincode::deserialize(&bytes)?;


CBOR? Just for cross language compatibility maybe?

I am still very undecided on the format, not a big fan of cbor, so for now this is a placeholder encoding

Newer clippy wants the identifiers in the format string when possible. Boring change but why not.

This makes sure to await the future from async_channel::Sender::send. When not awaiting this only makes a struct and noting is ever sent. Caught by clippy really.

Updates the requirements on [sysinfo](https://github.com/GuillaumeGomez/sysinfo) to permit the latest version. - [Release notes](https://github.com/GuillaumeGomez/sysinfo/releases) - [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md) - [Commits](https://github.com/GuillaumeGomez/sysinfo/commits) --- updated-dependencies: - dependency-name: sysinfo dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Floris Bruynooghe <[email protected]>

Updates the requirements on [rlimit](https://github.com/Nugine/rlimit) to permit the latest version. - [Release notes](https://github.com/Nugine/rlimit/releases) - [Changelog](https://github.com/Nugine/rlimit/blob/master/CHANGELOG.md) - [Commits](Nugine/rlimit@v0.8.3...v0.8.3) --- updated-dependencies: - dependency-name: rlimit dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Floris Bruynooghe <[email protected]>

Updates the requirements on [base64](https://github.com/marshallpierce/rust-base64) to permit the latest version. - [Release notes](https://github.com/marshallpierce/rust-base64/releases) - [Changelog](https://github.com/marshallpierce/rust-base64/blob/master/RELEASE-NOTES.md) - [Commits](marshallpierce/rust-base64@v0.13.1...v0.13.1) --- updated-dependencies: - dependency-name: base64 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Floris Bruynooghe <[email protected]>

The store does not use the RPC client at all. Remove it.

We used to use the same config for the service and server (aka the binary). This is confusing when creating configs to use with e.g. iroh-one, iroh-embed or iroh-share because some fields are not used. This splits off the config structs to avoid this problem, services now have a Config and binaries a ServerConfig. This allows creating the services standalone without all the baggage a server needs. While this isn't many fields yet, this will get worse as we add more features (this is split off from another PR where this seemed useful).

use quic & larger chunks

dignifiedquire and others added 30 commits October 15, 2022 00:08

reduce invalid stored connections

31cb29c

metrics: bitswap loop & poll ticks

a160805

feat: race a http gateway with p2p in iroh-one's content loader (#338)

40cb232

docs: add full help text for lookup, connect, get, p2p, and `…

b14a7a8

…iroh` commands (#348) * docs: add full help text for `lookup`, `connect`, `get`, `p2p`, and `iroh` commands * move long descriptions into constants Co-authored-by: b5 <[email protected]>

allow for more dialing out by default

4570621

refactor message queue actor

0904280

try: trigger connects for non bitswap peers

a95d0f5

improve gc and stopping of message queue

56f1df9

fix client loop metric

381f934

track peers

09a5155

improve peer state tracking

71eaab3

only skip real dials when dialing is paused

671d178

fixup triggers

a82ade2

improve connection management

8191cb7

more counters and rely on disconnects for cleanup

0afefe5

connect with identify

16c40c9

misc fixes and debugging

8e3f831

refactor: lock free peer manager

733601d

remove client unused client counters

cd78bd3

serialize incoming message processing

ced8e05

log backpressure for network actions

00a55c6

more evil

4bdec19

more detailed message metrics

a8303df

more debugging

6841992

getting closer

6046da1

bigger buffers

e4b6beb

do not wait for worker to shutdown

85435a2

handle responses only on a best effort

2e4de89

improve message scheduling

6f51330

implement connection protection

ebf39d5

fix(iroh): use local dep to avoid recursion

7a11aa8

hannahhoward reviewed Dec 28, 2022

View reviewed changes

flub and others added 5 commits January 2, 2023 12:35

chore(clippy): Place identifiers in format string (#663)

b7a4dd2

Newer clippy wants the identifiers in the format string when possible. Boring change but why not.

fix: Await async_channel::Sender::send (#662)

4011d3b

This makes sure to await the future from async_channel::Sender::send. When not awaiting this only makes a struct and noting is ever sent. Caught by clippy really.

dignifiedquire force-pushed the feat-memesync branch from 025c07c to 47ebcec Compare January 2, 2023 20:59

Arqu and others added 2 commits January 4, 2023 15:38

fix: force raw codec if requested (#644)

53b9220

ref(store): Remove unused RPC Client (#672)

1d30916

The store does not use the RPC client at all. Remove it.

dignifiedquire force-pushed the feat-memesync branch from 47ebcec to 2ec30a1 Compare January 4, 2023 17:22

flub and others added 10 commits January 4, 2023 20:07

[wip] iroh-memesync

670aca8

cleanup

f9ff073

try to stream responses

d6be392

recurse over thy blocks

3be2019

verify blocks and parse incoming links

103a667

integrate memesync into content loading

ceb0bc2

fixup rebase

a261d00

update deps

f5532f7

fixup rebase

0402f28

dignifiedquire force-pushed the feat-memesync branch from 2ec30a1 to 0402f28 Compare January 4, 2023 19:19

dignifiedquire added 2 commits January 4, 2023 21:46

fix(node): avoid spawning tasks for sending on channels

f0c9f94

improve iroh share

17e037d

use quic & larger chunks

dignifiedquire added this to the v0.3.0 milestone Jan 5, 2023

dignifiedquire mentioned this pull request Jan 6, 2023

Autonat + Quic Fails on Localhost libp2p/rust-libp2p#3308

Open

experiment: cbor

6a96391

dignifiedquire closed this Feb 20, 2023

dignifiedquire force-pushed the main branch from 6527ee4 to 3a3fc46 Compare February 20, 2023 15:47

dignifiedquire deleted the feat-memesync branch February 20, 2023 15:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wip] iroh-memesync #586

[wip] iroh-memesync #586

dignifiedquire commented Dec 12, 2022

hannahhoward left a comment

hannahhoward Dec 28, 2022

dignifiedquire Dec 31, 2022

hannahhoward Jan 4, 2023

hannahhoward Dec 28, 2022

dignifiedquire Dec 31, 2022

[wip] iroh-memesync #586

[wip] iroh-memesync #586

Conversation

dignifiedquire commented Dec 12, 2022

hannahhoward left a comment

Choose a reason for hiding this comment

hannahhoward Dec 28, 2022

Choose a reason for hiding this comment

dignifiedquire Dec 31, 2022

Choose a reason for hiding this comment

hannahhoward Jan 4, 2023

Choose a reason for hiding this comment

hannahhoward Dec 28, 2022

Choose a reason for hiding this comment

dignifiedquire Dec 31, 2022

Choose a reason for hiding this comment