From 8a05acc75898a099228d8b641dc631fb167329b4 Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Fri, 17 Feb 2023 21:48:34 +1300 Subject: [PATCH 1/6] Add initial yamux spec by copying in hashicorp one --- README.md | 2 + yamux/README.md | 147 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 149 insertions(+) create mode 100644 yamux/README.md diff --git a/README.md b/README.md index 68dfa8cf1..8155b8a65 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,7 @@ see [#465](https://github.com/libp2p/specs/issues/465). - [kademlia][spec_kademlia] - The Kademlia Distributed Hash Table (DHT) subsystem - [mdns][spec_mdns] - Local peer discovery with zero configuration using multicast DNS - [mplex][spec_mplex] - The friendly stream multiplexer +- [yamux][spec_yamux] - Yet Another Multiplexer - [noise][spec_noise] - The libp2p Noise handshake - [plaintext][spec_plaintext] - An insecure transport for non-production usage - [pnet][spec_pnet] - Private networking in libp2p using pre-shared keys @@ -142,3 +143,4 @@ you feel an issue isn't the appropriate place for your topic, please join our [spec_webrtc]: ./webrtc/README.md [spec_webtransport]: ./webtransport/README.md [spec_ping]: ./ping/ping.md +[spec_yamux]: ./yamux/README.md diff --git a/yamux/README.md b/yamux/README.md new file mode 100644 index 000000000..385d31cbb --- /dev/null +++ b/yamux/README.md @@ -0,0 +1,147 @@ +# yamux + +| Lifecycle Stage | Maturity | Status | Latest Revision | +|-----------------|----------------|--------|-----------------| +| 3A | Recommendation | Active | r0, 2023-02-17 | + +Authors: [@thomaseizinger] + +Interest Group: [@marten-seemann] + +[@thomaseizinger]: https://github.com/thomaseizinger +[@marten-seemann]: https://github.com/marten-seemann + +See the [lifecycle document][lifecycle-spec] for context about maturity level and spec status. + +[lifecycle-spec]: https://github.com/libp2p/specs/blob/master/00-framework-01-spec-lifecycle.md + +## Overview + +Yamux is Stream Multiplexer protocol originally specified by [@hashicorp](https://github.com/hashicorp). +The specification lives here: https://github.com/hashicorp/yamux/blob/master/spec.md + +The below sections are a verbatim copy (modulo formatting changes) of the spec at the time this document was created. +This allows us to preserve the specification in case the linked document is ever removed or edited. + +## Specification + +### Framing + +Yamux uses a streaming connection underneath, but imposes a message framing so that it can be shared between many logical streams. +Each frame contains a header like: + +* Version (8 bits) +* Type (8 bits) +* Flags (16 bits) +* StreamID (32 bits) +* Length (32 bits) + +This means that each header has a 12 byte overhead. +All fields are encoded in network order (big endian). +Each field is described below: + +#### Version Field + +The version field is used for future backward compatibility. +At the current time, the field is always set to 0, to indicate the initial version. + +#### Type Field + +The type field is used to switch the frame message type. +The following message types are supported: + +* 0x0 Data - Used to transmit data. May transmit zero length payloads depending on the flags. + +* 0x1 Window Update - Used to updated the senders receive window size. + This is used to implement per-session flow control. + +* 0x2 Ping - Used to measure RTT. + It can also be used to heart-beat and do keep-alives over TCP. + +* 0x3 Go Away - Used to close a session. + +#### Flag Field + +The flags field is used to provide additional information related to the message type. +The following flags are supported: + +* 0x1 SYN - Signals the start of a new stream. + May be sent with a data or window update message. + Also sent with a ping to indicate outbound. + +* 0x2 ACK - Acknowledges the start of a new stream. + May be sent with a data or window update message. + Also sent with a ping to indicate response. + +* 0x4 FIN - Performs a half-close of a stream. + May be sent with a data message or window update. + +* 0x8 RST - Reset a stream immediately. + May be sent with a data or window update message. + +#### StreamID Field + +The StreamID field is used to identify the logical stream the frame is addressing. +The client side should use odd ID's, and the server even. +This prevents any collisions. +Additionally, the 0 ID is reserved to represent the session. + +Both Ping and Go Away messages should always use the 0 StreamID. + +#### Length Field + +The meaning of the length field depends on the message type: + +* Data - provides the length of bytes following the header +* Window update - provides a delta update to the window size +* Ping - Contains an opaque value, echoed back +* Go Away - Contains an error code + +### Message Flow + +There is no explicit connection setup, as Yamux relies on an underlying transport to be provided. +However, there is a distinction between client and server side of the connection. + +#### Opening a stream + +To open a stream, an initial data or window update frame is sent with a new StreamID. +The SYN flag should be set to signal a new stream. + +The receiver must then reply with either a data or window update frame with the StreamID along with the ACK flag to accept the stream or with the RST flag to reject the stream. + +Because we are relying on the reliable stream underneath, a connection can begin sending data once the SYN flag is sent. +The corresponding ACK does not need to be received. +This is particularly well suited for an RPC system where a client wants to open a stream and immediately fire a request without waiting for the RTT of the ACK. + +This does introduce the possibility of a connection being rejected after data has been sent already. +This is a slight semantic difference from TCP, where the conection cannot be refused after it is opened. +Clients should be prepared to handle this by checking for an error that indicates a RST was received. + +#### Closing a stream + +To close a stream, either side sends a data or window update frame along with the FIN flag. +This does a half-close indicating the sender will send no further data. + +Once both sides have closed the connection, the stream is closed. + +Alternatively, if an error occurs, the RST flag can be used to hard close a stream immediately. + +#### Flow Control + +When Yamux is initially starts each stream with a 256KB window size. +There is no window size for the session. + +To prevent the streams from stalling, window update frames should be sent regularly. +Yamux can be configured to provide a larger limit for windows sizes. +Both sides assume the initial 256KB window, but can immediately send a window update as part of the SYN/ACK indicating a larger window. + +Both sides should track the number of bytes sent in Data frames only, as only they are tracked as part of the window size. + +#### Session termination + +When a session is being terminated, the Go Away message should be sent. +The Length should be set to one of the following to provide an error code: + +* 0x0 Normal termination +* 0x1 Protocol error +* 0x2 Internal error From 4e3ffc1016cfd09429788b077d425a97171d2492 Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Fri, 17 Feb 2023 21:51:58 +1300 Subject: [PATCH 2/6] Add ACK backlog recommendation --- yamux/README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/yamux/README.md b/yamux/README.md index 385d31cbb..d4b391687 100644 --- a/yamux/README.md +++ b/yamux/README.md @@ -145,3 +145,11 @@ The Length should be set to one of the following to provide an error code: * 0x0 Normal termination * 0x1 Protocol error * 0x2 Internal error + +## Implementation considerations + +### ACK backlog + +Yamux allows for a stream to be opened (and used) before it is acknowledged by the remote. +We defined the ACK backlog as the number of streams that a peer has opened which have not yet been acknowledged. +Implementations SHOULD at most allow an ACK backlog of 256 streams. From 6aaf9aa2f7722aeda90a72a88fe7b2ba6ef4bb48 Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Fri, 17 Feb 2023 21:55:04 +1300 Subject: [PATCH 3/6] Add wemeetagain to interest group --- yamux/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/yamux/README.md b/yamux/README.md index d4b391687..bc829075a 100644 --- a/yamux/README.md +++ b/yamux/README.md @@ -6,10 +6,11 @@ Authors: [@thomaseizinger] -Interest Group: [@marten-seemann] +Interest Group: [@marten-seemann], [@wemeetagain] [@thomaseizinger]: https://github.com/thomaseizinger [@marten-seemann]: https://github.com/marten-seemann +[@wemeetagain]: https://github.com/wemeetagain See the [lifecycle document][lifecycle-spec] for context about maturity level and spec status. From cee5bd0268b33c8640e9b8f5b1a4534b36b09e7f Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Sun, 26 Feb 2023 17:59:11 +1100 Subject: [PATCH 4/6] Apply suggestions from code review Co-authored-by: Marten Seemann --- yamux/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/yamux/README.md b/yamux/README.md index bc829075a..ac153164f 100644 --- a/yamux/README.md +++ b/yamux/README.md @@ -115,7 +115,7 @@ The corresponding ACK does not need to be received. This is particularly well suited for an RPC system where a client wants to open a stream and immediately fire a request without waiting for the RTT of the ACK. This does introduce the possibility of a connection being rejected after data has been sent already. -This is a slight semantic difference from TCP, where the conection cannot be refused after it is opened. +This is a slight semantic difference from TCP, where the connection cannot be refused after it is opened. Clients should be prepared to handle this by checking for an error that indicates a RST was received. #### Closing a stream @@ -152,5 +152,5 @@ The Length should be set to one of the following to provide an error code: ### ACK backlog Yamux allows for a stream to be opened (and used) before it is acknowledged by the remote. -We defined the ACK backlog as the number of streams that a peer has opened which have not yet been acknowledged. +The ACK backlog is defined as the number of streams that a peer has opened which have not yet been acknowledged. Implementations SHOULD at most allow an ACK backlog of 256 streams. From 3b2ec80eddb1f8f03c0d3472aeebb430151165b8 Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Sun, 26 Feb 2023 17:57:53 +1100 Subject: [PATCH 5/6] Add yamux protocol string --- yamux/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/yamux/README.md b/yamux/README.md index ac153164f..31444d9f2 100644 --- a/yamux/README.md +++ b/yamux/README.md @@ -26,6 +26,8 @@ This allows us to preserve the specification in case the linked document is ever ## Specification +The protocol string of yamux for `multistream-select` is: `/yamux/1.0.0`. + ### Framing Yamux uses a streaming connection underneath, but imposes a message framing so that it can be shared between many logical streams. From c19d06b0473ad104aa622cda6ca11bbc5f2f04e2 Mon Sep 17 00:00:00 2001 From: Thomas Eizinger Date: Sun, 26 Feb 2023 17:59:26 +1100 Subject: [PATCH 6/6] Add ianopolous to interest group --- yamux/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/yamux/README.md b/yamux/README.md index 31444d9f2..efab67b57 100644 --- a/yamux/README.md +++ b/yamux/README.md @@ -6,11 +6,12 @@ Authors: [@thomaseizinger] -Interest Group: [@marten-seemann], [@wemeetagain] +Interest Group: [@marten-seemann], [@wemeetagain], [@ianopolous] [@thomaseizinger]: https://github.com/thomaseizinger [@marten-seemann]: https://github.com/marten-seemann [@wemeetagain]: https://github.com/wemeetagain +[@ianopolous]: https://github.com/ianopolous See the [lifecycle document][lifecycle-spec] for context about maturity level and spec status.