Make more clear how we use the block header to calculate the txs short ids (job declarator)

[Fi3]

In SRI we do not use the block header to calculate the txs short ids (job declarator). This make sense since we do not know the block header yet. So maybe we should change the specification and make it clear, or use the previous block header? I would go with the former since is easier

[GitGab19]

I recovered the way we are computing it on SRI and I confirm we're currently only using the SHA256(nonce) as key of the SipHash.
I tried to think about using the previous block header, but it would add complexity and since it would be known, it would be predictable as well (so I don't think it would add more security). What would happen in the case in which JDS and JDC are seeing two different tips of the chain? It would only add a bit more resistance to collisions I guess.

I would go with specifying better the only usage of the nonce.

Maybe @TheBlueMatt could have some other hints here?

[TheBlueMatt]

Yea, there's no reason to want the block header there, I think. We honestly could also skip the SHA256 but given people have already implemented it no reason to drop it.

[Fi3]

Yea, there's no reason to want the block header there, I think. We honestly could also skip the SHA256 but given people have already implemented it no reason to drop it.

Agree about the SHA256, and I would be ok in changing spec. But maybe now is too late

[GitGab19]

I would just make this point in specs more clear, without changing too much the way the short_tx_id is specified in BIP 152

[Fi3]

ok with me

[Fi3]

I'm ok with removing the sha cause is computationally very expensive, but also changing the spec at this point is not very nice

[Fi3]

so whatever I don't have a preference

[GitGab19]

@jakubtrnka would it be a problem for you to remove the SHA256 from the short_tx_id computation?

[GitGab19]

Considering the removal of SHA256, we are still going to need a 128-bit key for the SipHash.
Do you think it would be better to have a larger nonce (currently it's a U64, it could become a U256), or adding a second nonce (always a U64)? @Fi3 @TheBlueMatt

[Fi3]

Considering the removal of SHA256, we are still going to need a 128-bit key for the SipHash. Do you think it would be better to have a larger nonce (currently it's a U64, it could become a U256), or adding a second nonce (always a U64)? @Fi3 @TheBlueMatt

I would just double the u64. So if you have nonce = 10. You get, key = 10.to_le_bytes() ++ 10.to_le_bytes()

[TheBlueMatt]

I liked the DATUM idea of using the two peer's ephemeral noise keys (or some other pet-connection random value which both peers contribute to).

[Fi3]

I liked the DATUM idea of using the two peer's ephemeral noise keys (or some other pet-connection random value which both peers contribute to).

I prefer having some form of "control" over the nonce. I'm planning to create an extension where pool can tell miners which nonce to use. I want to cache short ids and not having to calculate them for each client. So I nack this proposal.

@TheBlueMatt anything wrong in the below proposal?

I would just double the u64. So if you have nonce = 10. You get, key = 10.to_le_bytes() ++ 10.to_le_bytes()

[Fi3]

Just to add on my previous comment I want to say that I'm not making this up, I ran several bench tests to the pool and the bootleneck on adding new clients is always calculating short ids.

[TheBlueMatt]

Calculating them on a per-client basis shouldn't be too bad if you cache it, no? The current spec requires you calculate them live when you get a new job and I agree that is too conservative. However, having a fixed value for the entire pool makes it trivial for anyone to attack all the pool's connections at once just for the lulz.

[TheBlueMatt]

If even that's too much I think we should just send the first 160/192 bits of the txid over the wire instead.

[Fi3]

Calculating them on a per-client basis shouldn't be too bad if you cache it, no? The current spec requires you calculate them live when you get a new job and I agree that is too conservative. However, having a fixed value for the entire pool makes it trivial for anyone to attack all the pool's connections at once just for the lulz.

I don't know how much it save to remove the sha256, maybe is enough. About the attack yes and no, I can always find who is doing it and ban him pretty fast log2(users) steps, this only if I have registered users, if anyone can join would be a problem. So I don't' know. Just to recap here the options:

1. send 160/192 bits
2. use a secrete that only pool and client know
3. pool can ask client to change the nonce with an extension an ad hoc message

Personally, I would go with 2 or 3. With 3 you can implement 2.

[Fi3]

also not having to do that live is a great improvement in general

[TheBlueMatt]

I can always find who is doing it and ban him pretty fast log2(users) steps

Sure, but are you really gonna implement that? That's a lot of work and I'm pretty sure most pools won't bother. And, indeed, we don't really want to build a protocol that requires pools not be open to anyone.

Fwiw I'm somewhat skeptical that generating short IDs for transactions in the mempool is a huge performance penalty, IIRC Bitcoin Core can do it in a handful of ms IIRC. What are you seeing?

[Fi3]

If you have 1000 clients an handfull of ms are too much. You can calculate them when you find the block lets say that you calculate 1000 * 2000 -> 2_000_000 short ids when you find the block and the rest while the tx arrives. Is 10 seconds. You can use 10 cores and do that in 1 seconds. Is kinda of acceptable but not the best. I also I don't need 10 cpu for 1000 clients.

[Fi3]

maybe you meant mircroseconds?

[TheBlueMatt]

If you just calculate once per connection (and then in the background) it's not that crazy? You'd have to authenticate clients somehow (which we should probably do anyway, random connections negotiating a million jobs without any hashpower isn't really acceptable either). A restart is a bit painful, but only for a second or two? That's not crazy.

[Fi3]

If you just calculate once per connection (and then in the background) it's not that crazy? You'd have to authenticate clients somehow (which we should probably do anyway, random connections negotiating a million jobs without any hashpower isn't really acceptable either). A restart is a bit painful, but only for a second or two? That's not crazy.

demand offer a free solo pool (separated from the actual pool) with job declaration enabled and is completely anonymous so no auth. Btw yes is not crazy.

[GitGab19]

Calculating them on a per-client basis shouldn't be too bad if you cache it, no? The current spec requires you calculate them live when you get a new job and I agree that is too conservative. However, having a fixed value for the entire pool makes it trivial for anyone to attack all the pool's connections at once just for the lulz.

I don't know how much it save to remove the sha256, maybe is enough. About the attack yes and no, I can always find who is doing it and ban him pretty fast log2(users) steps, this only if I have registered users, if anyone can join would be a problem. So I don't' know. Just to recap here the options:

1. send 160/192 bits

2. use a secrete that only pool and client know

3. pool can ask client to change the nonce with an extension an ad hoc message

Personally, I would go with 2 or 3. With 3 you can implement 2.

I would go with 3, using the AllocateMiningJobToken.Success to send the Siphash keys to JDC.

[TheBlueMatt]

I'm very afraid of a protocol that enables the pool to select the nonce, as I imagine most pools will use it to simply give every client a static nonce. If we're gonna go that way we should just remove the hashing entirely and send txids/short txids.

IOW, I object to 3 on the grounds that it enables dangerous behavior. Thus if we think the performance of 2 is bad, we should just do 1, which honestly isn't that bad.

[plebhash]

assuming that the main goal is simply to save bandwidth on DeclareMiningJob, truncating txids and only sending the initial bits seems quite straightforward and honestly quite appealing

but I don't fully understand what is the rationale behind 160/192 bits:

• when we say 160/192, do we mean either 160 or 192 bits?
• why these specific numbers? are they some kind of redundancy threshold for avoiding collisions?
• why do we have two options on the table? what are the tradeoffs of choosing one over the other?

[Fi3]

assuming that the main goal is simply to save bandwidth on DeclareMiningJob, truncating txids and only sending the initial bits seems quite straightforward and honestly quite appealing

a short id is a lot more short, but is ok with me, don't know if it cost more the bandwidth or the cpu time. In that case maybe bandwidth. But I don't thing that the difference is so big, maybe we should wrote down some number.

when we say 160/192, do we mean either 160 or 192 bits?

I guess we should stick with one? Or you maybe you can require less bit and then more?

why these specific numbers? are they some kind of redundancy threshold for avoiding collisions?

is to avoid that forging collision (attacking the pool) become too easy

why do we have two options on the table? what are the tradeoffs of choosing one over the other?

tradeoff are cpu time vs bandwidth

[Fi3]

Ok so as we agreed on the discord chat we are going to remove short ids from JD protocol, and just send full ids. When needed we can reintroduce short ids trough an extension.

@TheBlueMatt @GitGab19 @plebhash do you confirm?

[TheBlueMatt]

Yep!