btcec update leads to go mod tidy failure

[marten-seemann]

We just released v0.16.0, which include the btcec update (#245).

go mod tidy now produces the following error (see libp2p/go-libp2p#1521):

❯ go mod tidy
github.com/libp2p/go-libp2p imports
        github.com/libp2p/go-libp2p-core/crypto imports
        github.com/btcsuite/btcd/btcec/v2/ecdsa tested by
        github.com/btcsuite/btcd/btcec/v2/ecdsa.test imports
        github.com/btcsuite/btcd/chaincfg/chainhash loaded from github.com/btcsuite/[email protected],
        but go 1.16 would fail to locate it:
        ambiguous import: found package github.com/btcsuite/btcd/chaincfg/chainhash in multiple modules:
        github.com/btcsuite/btcd v0.22.0-beta (/Users/marten/src/go/pkg/mod/github.com/btcsuite/[email protected]/chaincfg/chainhash)
        github.com/btcsuite/btcd/chaincfg/chainhash v1.0.0 (/Users/marten/src/go/pkg/mod/github.com/btcsuite/btcd/chaincfg/[email protected])

To proceed despite packages unresolved in go 1.16:
        go mod tidy -e
If reproducibility with go 1.16 is not needed:
        go mod tidy -compat=1.17
For other options, see:
        https://golang.org/doc/modules/pruning

Updating btced to v2.2.0 (from v2.1.3) doesn't solve the problem. I also haven't succeeded in manually adding github.com/btcsuite/btcd/chaincfg/chainhash to go-libp2p's go.mod file.
People had a similar issue in btcsuite/btcd#1839, but I couldn't make it work in go-libp2p either.

This is now blocking the go-libp2p v0.20.0 release. I'm planning to revert #245, unless we can get this solved within the next 24 hours or so.

cc @brianathere

[brianathere]

Looking at this now.

[vyzo]

Can we fix our CI globally to do 1.17 compat tidy?
We've had this problem elsewhere too.

[marten-seemann]

Can we fix our CI globally to do 1.17 compat tidy?

We could, but the problem is that every user of go-libp2p would not be able to run a normal go mod tidy any more.

I don't really understand why the Go tool would care about 1.16 when the go.mod file clearly states go 1.17 (as all our go.mod files do), but apparently that's what the Go team decided.

[galargh]

I found another change that makes the errors go away - see libp2p/go-libp2p#1524 - but I'm yet to understand exactly what's going on.

[marten-seemann]

Adding the following file to go-libp2p works:

//go:build !go1.17
// +build !go1.17

package libp2p

import (
	_ "github.com/btcsuite/btcd/chaincfg"
)

and then making sure that go.mod has require github.com/btcsuite/btcd v0.22.1.

[marten-seemann]

This could also be applied in this repo: #254

Not sure where the right place for this to live would be. Thoughts?

[galargh]

Given what I've read so far here's my guess.

Before

go-libp2p depends on [email protected] and [email protected].

github.com/libp2p/go-libp2p github.com/btcsuite/[email protected]
github.com/libp2p/[email protected] github.com/btcsuite/[email protected]
github.com/libp2p/[email protected] github.com/btcsuite/[email protected]
github.com/libp2p/[email protected] github.com/btcsuite/[email protected]
github.com/libp2p/[email protected] github.com/btcsuite/[email protected]
github.com/libp2p/[email protected] github.com/btcsuite/[email protected]
github.com/libp2p/[email protected] github.com/btcsuite/[email protected]

This is fine, these two are compatible (in terms of what go mod tidy does i.e. dep resolution model of go 1.16) with each other.

The only version of chainhash that is used is the one that comes as a subdir of btcd.

After

go-libp2p-core changes dependency on [email protected] to a dependency on btcd/btcec/v2 v2.1.3. Important note: btcec depends on released chainhash (v1.0.0 to be exact).

go-libp2p still depends on [email protected] and [email protected].

github.com/libp2p/go-libp2p github.com/btcsuite/[email protected]
github.com/libp2p/[email protected] github.com/btcsuite/[email protected]
github.com/libp2p/[email protected] github.com/btcsuite/[email protected]
github.com/libp2p/[email protected] github.com/btcsuite/[email protected]
github.com/libp2p/[email protected] github.com/btcsuite/[email protected]
github.com/libp2p/[email protected] github.com/btcsuite/[email protected]

But now it also depends on [email protected].

github.com/libp2p/[email protected] github.com/btcsuite/btcd/btcec/[email protected]

And transitively, it also starts depending on released [email protected].

github.com/btcsuite/btcd/btcec/[email protected] github.com/btcsuite/btcd/chaincfg/[email protected]

Now we have 2 incompatible versions of chainhash in chain, one from btcd's subdir and one from a proper v1.0.0 release.

Resolution

We can either force replace btcd to v0.22.1 which doesn't have chainhash subdir at all anymore, or we can upgrade btcd in all of these:

• libp2p/go-libp2p
• libp2p/go-libp2p-quic-transport
• libp2p/go-libp2p-resource-manager
• libp2p/go-libp2p-testing
• libp2p/go-libp2p-tls
• libp2p/go-libp2p-yamux

The latter seems cleaner but it's much more work.

[galargh]

This could also be applied in this repo: #254

Not sure where the right place for this to live would be. Thoughts?

Nice! I tested it locally and it seems to do the trick :)

Testing

I checked out your branch in go-libp2p-core and updated the reference in go-libp2p's go.mod to point at the local checkout. After this, I ran go mod tidy and it didn't complain anymore.

[galargh]

I think maybe #252 (comment) fits better in go-libp2p because we'd be able to remove it once/if we stop transitively depending on lower versions?

Either way, I think yours is better than my replace all.