-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(net): support eth/68 #1361
Conversation
Codecov Report
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more @@ Coverage Diff @@
## main #1361 +/- ##
===========================================
- Coverage 76.19% 47.27% -28.93%
===========================================
Files 357 358 +1
Lines 41040 42320 +1280
===========================================
- Hits 31270 20005 -11265
- Misses 9770 22315 +12545
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for taking this on.
left some comments, but would like to hear @Rjected s opinion on some parts.
NewPooledTransactionHashes(NewPooledTransactionHashes), | ||
NewPooledTransactionHashes68(NewPooledTransactionHashes68), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is a bit cursed -.- that the variant now is an enum
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I think given we don't have support for multiple capabilities, which also has the need for new message types and encodings, it's up in the air how we should do this properly. But yeah, I agree this doesn't seem like the best way to do it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree it's too ugly and immature.
I adopted the convention from geth
. (They uses the name like NewPooledTransactionHashesPacket66
and NewPooledTransactionHashesPacket68
.) And I tried to minimize changes. This code is the result of this approach.
I'll think more what's better way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I actually like that we do version-specific decoding on the ethstream level.
so, after some thinking this change seems to be fine to me, because we already have outdated variants like GetNodeData etc.
and because the stream does not emit them if they don't match the version this is ok.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to rename the existing variant to NewPooledTransactionHashes66
then
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to rename the existing variant to NewPooledTransactionHashes66 then
That's my first implementation but I've reverted it. I'll think again and share why I reverted it. Or will follow up the idea.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm learning for ActiveSession
, PeerMessage::PooledTransactions
and so on. The PR might have a bug.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After learning code more, I could remember why I choose to use NewPooledTransactionHashes
than NewPooledTransactionHashes66
. It's because NewPooledTransactionHashes
is used internally as well for PeerMessage::PooledTransactions
, NetworkHandleMessage::SendPooledTransactionHashes
and NetworkTransactionEvent::IncomingPooledTransactionHashes
.
Now I've realized that we should care of these as well. maybe need more 66
, 68
convention and branches. T_T I'm working on it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to rename the existing variant to NewPooledTransactionHashes66 then
follow it up with a94b1da. I've changed this PR into Draft because the new change is little bit big. I need to add more tests and review myself first.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ready for review.
@@ -181,6 +181,19 @@ impl ActiveSession { | |||
EthMessage::NewPooledTransactionHashes(msg) => { | |||
self.try_emit_broadcast(PeerMessage::PooledTransactions(msg)).into() | |||
} | |||
EthMessage::NewPooledTransactionHashes68(msg) => { | |||
if msg.hashes.len() != msg.types.len() || msg.hashes.len() != msg.sizes.len() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Rjected perhaps we should apply this check in the stream directly? so the stream only yields well-formatted messages
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, we should be doing these checks in the stream, and malformed messages should be considered invalid
message_type: EthMessageID, | ||
buf: &mut &[u8], | ||
) -> Result<Self, reth_rlp::DecodeError> { | ||
pub fn decode_message(version: u8, buf: &mut &[u8]) -> Result<Self, reth_rlp::DecodeError> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so now decode_message depends on the version? because NewPooledTransactionHashes
format differs?
...
I feel like mixing this together with decode is a bit weird. for example decoding GetNodeData now fails with an rlp error even though it can still be valid rlp.
I really would like to do this in two steps:
decode then validate against version.
wdyt @Rjected
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, decoding should not worry about version details, I think this is another consequence of not supporting multiple capabilities. We don't know what to decode into if we don't know the eth wire protocol version being used. It's ambiguous whether or not we should attempt to decode a eth/68
pooled hashes message, or a eth/67
pooled hashes message when we receive a NewPooledTransactionHashes
message ID.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For support multiple capabilities, we must have a branch at some level. It could be just if
, match
, traits
, module
or whatever. It's needed but I agree that it's not the best way.
crates/net/eth-wire/src/hello.rs
Outdated
capabilities: capabilities | ||
.unwrap_or_else(|| vec![EthVersion::Eth66.into(), EthVersion::Eth67.into()]), | ||
capabilities: capabilities.unwrap_or_else(|| { | ||
vec![EthVersion::Eth66.into(), EthVersion::Eth67.into(), EthVersion::Eth68.into()] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we should enable this by default already?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agreed, this is a future upgrade, 67
is still considered the current version
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agreed. I thought it's safe because it's still including 66
, 67
, but I was too naive and aggressive. Definitely, I agree not to enable it yet and need to verify it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
follow it up with 651fc4d
crates/net/eth-wire/src/ethstream.rs
Outdated
#[pin] | ||
inner: S, | ||
} | ||
|
||
impl<S> EthStream<S> { | ||
/// Creates a new unauthed [`EthStream`] from a provided stream. You will need | ||
/// to manually handshake a peer. | ||
pub fn new(inner: S) -> Self { | ||
Self { inner } | ||
pub fn new(version: u8, inner: S) -> Self { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if we do this then all version: u8
should be version: EthVersion
instead
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
follow it up with ca74056
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm really not sure that we should implement this without first figuring out the best way to support multiple capabilities. eth/66
and eth/67
are a convenient exception to the rule that most versions have entirely separate message encodings, that are ambiguous unless you know the version. New capabilities also have this problem, so if we push to support eth/68
without first figuring out how to support other capabilities, I think we might bury a lot of version-specific code / tech debt that should not be there long-term
NewPooledTransactionHashes(NewPooledTransactionHashes), | ||
NewPooledTransactionHashes68(NewPooledTransactionHashes68), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I think given we don't have support for multiple capabilities, which also has the need for new message types and encodings, it's up in the air how we should do this properly. But yeah, I agree this doesn't seem like the best way to do it
@@ -181,6 +181,19 @@ impl ActiveSession { | |||
EthMessage::NewPooledTransactionHashes(msg) => { | |||
self.try_emit_broadcast(PeerMessage::PooledTransactions(msg)).into() | |||
} | |||
EthMessage::NewPooledTransactionHashes68(msg) => { | |||
if msg.hashes.len() != msg.types.len() || msg.hashes.len() != msg.sizes.len() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, we should be doing these checks in the stream, and malformed messages should be considered invalid
message_type: EthMessageID, | ||
buf: &mut &[u8], | ||
) -> Result<Self, reth_rlp::DecodeError> { | ||
pub fn decode_message(version: u8, buf: &mut &[u8]) -> Result<Self, reth_rlp::DecodeError> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, decoding should not worry about version details, I think this is another consequence of not supporting multiple capabilities. We don't know what to decode into if we don't know the eth wire protocol version being used. It's ambiguous whether or not we should attempt to decode a eth/68
pooled hashes message, or a eth/67
pooled hashes message when we receive a NewPooledTransactionHashes
message ID.
Thank you for comments! I'm much learning from your comments. And now I understand it could cause tech debt and we need to figure out what's the best way to support multiple protocol. Please feel free to close it if this PR is not proper way. I just picked it up because the issue is marked as |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
alright, we actually need this urgently because this is now a blocker for hive tests.
This looks pretty good already, I guess biggest change is: move tx hash 68 validation into stream itself.
and separate decoding from version but the stream should validate the decoded message against the version, right @Rjected ?
@@ -115,7 +122,7 @@ impl Capabilities { | |||
/// Whether the peer supports `eth` sub-protocol. | |||
#[inline] | |||
pub fn supports_eth(&self) -> bool { | |||
self.eth_67 || self.eth_66 | |||
self.eth_67 || self.eth_66 || self.eth_68 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since we expect that client will update over time we can change the order here: 68 || 67 || 66
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will do
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
follow it up with b7b4b14. nit but please note that I also changed the eth versions order as desc in HelloMessageBuilder
.
NewPooledTransactionHashes(NewPooledTransactionHashes), | ||
NewPooledTransactionHashes68(NewPooledTransactionHashes68), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I actually like that we do version-specific decoding on the ethstream level.
so, after some thinking this change seems to be fine to me, because we already have outdated variants like GetNodeData etc.
and because the stream does not emit them if they don't match the version this is ok.
NewPooledTransactionHashes(NewPooledTransactionHashes), | ||
NewPooledTransactionHashes68(NewPooledTransactionHashes68), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to rename the existing variant to NewPooledTransactionHashes66
then
@@ -153,6 +158,7 @@ pub enum EthMessage { | |||
NewBlock(Box<NewBlock>), | |||
Transactions(Transactions), | |||
NewPooledTransactionHashes(NewPooledTransactionHashes), | |||
NewPooledTransactionHashes68(NewPooledTransactionHashes68), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also need to update struct docs for EthMessage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
follow it up with 011eb73. Please suggest if docs is insufficient because I'm not a fluent English speaker.
I guess it might be difficult to decode without using the version, because otherwise we don't know if we should attempt a |
merged |
I need to investigate why the test is failed because it seems not to be related with my change. Please note that the work might be delayed because it's weekend in my timezone. |
Co-authored-by: Georgios Konstantopoulos <[email protected]>
…and follow it up
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good work! I think final things missing might be some tests? Defer to @mattsse otherwise. I guess passing Hive will matter there, but worth configuring 2 clients with eth/68 and seeing that it works? Or one with 67 and one with 68?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
only a few suggestion for the eth-wire changes.
I have some questions regarding the network changes, mainly why we need this type with optional fields.
can we extract all of these and submit separately, so we can merge the eth68 changes asap?
EthHandshakeError(#[from] EthHandshakeError), | ||
#[error("message size ({0}) exceeds max length (10MB)")] | ||
MessageTooBig(usize), | ||
#[error("TransactionHashes invalid len of fields: {0} {1} {2}")] | ||
TransactionHashesInvalidLenOfFields(usize, usize, usize), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd like to use named fields here, otherwise it's harder to understand what the different usizes are
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
follow it up with 231f460.
pub hashes: Vec<H256>, | ||
} | ||
|
||
impl TryFrom<(Option<Vec<u8>>, Option<Vec<usize>>, Vec<H256>)> for NewPooledTransactionHashes68 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not a fan of this conversion.
why would we need this anyway?
I'd like to remove this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please give me an advice. T_T
- It's used for converting from
NewPooledTransactionHashes
intoNewPooledTransactionHashes68
. NewPooledTransactionHashes
is defined innetwork
so we could not defineTryFrom<NewPooledTransactionHashes> for NewPooledTransactionHashes68
.- The only alternative idea I have is that
pub fn new(...) -> Result<Self, Error>
.
HDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need this in this PR?
I still don't understand why we would need this.
in any case, a TryFrom a tuple with 3 items is not great.
if version >= EthVersion::Eth67 { | ||
return Err(reth_rlp::DecodeError::Custom("invalid message id")) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think since this is no longer rlp only we need to change the error type we return as well and add a dedicated error variant for when this happens.
crates/net/network/src/message.rs
Outdated
/// Internal form of a `PooledTransactions` message | ||
/// Same as [`NewPooledTransactionHashes68`] but types and sizes are Option. | ||
#[derive(Debug, Clone)] | ||
pub struct NewPooledTransactionHashes { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do we need this, why do we need optional fields here?
crates/net/network/src/message.rs
Outdated
} | ||
} | ||
|
||
impl From<(Vec<TxHash>, Vec<TxType>, Vec<usize>)> for NewPooledTransactionHashes { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From impls for tuples are not great so would abandon them entirely.
but not clear why we need this type at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
follow it up with acb0d81.
I've tested and succeeded it with enabling only eth/68 and connecting to |
Thanks for your comment! To my understanding, maybe you asked why does reth/crates/net/network/src/message.rs Lines 40 to 59 in 058349e
My first approach was decomposing I wonder whether it's acceptable or not. Always welcome for any advice. |
I see, I'd like to add another variant for 68 here as well, but let's do that separately, we can leave incoming 68 unhandled in this PR |
@jinsankim can we please do the bare minimum for eth68 here and do the new message type in a separate PR? |
little bit confusing. I need more guide about what's the bare minimum for eth68. I agree whatever we want for this PR, close, modify, revert only some part or whatever. But I can't catch what should I do for bare minimum. |
crates/net/network/src/message.rs
Outdated
/// Internal form of a `PooledTransactions` message | ||
/// Same as [`NewPooledTransactionHashes68`] but types and sizes are Option. | ||
#[derive(Debug, Clone)] | ||
pub struct NewPooledTransactionHashes { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please remove this type and instead move to separate PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
follow it up with f53e2a7.
pub hashes: Vec<H256>, | ||
} | ||
|
||
impl TryFrom<(Option<Vec<u8>>, Option<Vec<usize>>, Vec<H256>)> for NewPooledTransactionHashes68 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please remove this trait impl entirely.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
follow it up with f53e2a7.
What I've caught your direction from 2 todos,
Am I understanding correctly? |
yeh, I want to merge this with just 68 support on the eth wire level to unblock hive tests. |
I have no knowledge for hive tests. But I've just concerned that if we send I agree to merge PR step by step! |
* minimize changes * remove `NewPooledTransactionHashes`
Now, in this PR, it could handle incoming eth/68 message but couldn't handle outgoing eth/68 message. I left a TODO comments for it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mattsse calling this good, the logic for parsing msgs seems fine.
main 2 things are:
- We do
try_emit_broadcast
w/o types/sizes toNewPooledTransactionHashes68
msgs, is this correct? Or will we get penalized? - The TODO at active.rs#Implement new transaction propagation #254
I think the incoming handling will be working well.
outgoing is removed from this PR because of unacceptable message structure. I hope it'll be followed up with #254 and also improve architecture for multi protocol in the future. |
I think it's too weird. :( |
Closes #1326
Motivation
Reference