Skip to content
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.

remove replace directives #2590

Merged
merged 1 commit into from
Nov 14, 2019
Merged

remove replace directives #2590

merged 1 commit into from
Nov 14, 2019

Conversation

carnott-snap
Copy link
Contributor

@carnott-snap carnott-snap commented Nov 7, 2019

go.mod contains many replace entries that all consumers must include to
import logic. These are cancerous and tend to make consumers builds
difficult or outright fail. All have been removed, except for the one
required by helm-operator. This entry, while retained, has been
converted into a noop and should be removed when
fluxcd/helm-operator#97 is fixed.

carnott-snap added a commit to carnott-snap/helm-operator that referenced this pull request Nov 7, 2019
go.mod contains many replace entries that all consumers must include to
import logic. These are cancerous and tend to make consumers builds
difficult or outright fail. All have been removed, except for the one
required by flux. This entry, while retained, has converted into a noop
and should be removed when fluxcd/flux#2590 is fixed.
carnott-snap added a commit to carnott-snap/helm-operator that referenced this pull request Nov 7, 2019
go.mod contains many replace entries that all consumers must include to
import logic. These are cancerous and tend to make consumers builds
difficult or outright fail. All have been removed, except for the one
required by flux. This entry, while retained, has been converted into a
noop and should be removed when fluxcd/flux#2590 is fixed.
carnott-snap added a commit to carnott-snap/helm-operator that referenced this pull request Nov 7, 2019
go.mod contains many replace entries that all consumers must include to
import logic. These are cancerous and tend to make consumers builds
difficult or outright fail. All have been removed, except for the one
required by flux. This entry, while retained, has been converted into a
noop and should be removed when fluxcd/flux#2590 is fixed.
@2opremio
Copy link
Contributor

2opremio commented Nov 7, 2019

Hi. Thanks for the contribution!

However, I am afraid that those replacements are needed. What makes you think they are not?

It may be that flux even compiles without them but that doesn't mean they are not needed.

These are cancerous and tend to make consumers builds difficult or outright fail

I would generally agree with that, but, they are needed. Also, could you provided an existing scenario in which the replacements are an issue?

carnott-snap added a commit to carnott-snap/helm-operator that referenced this pull request Nov 7, 2019
go.mod contains many replace entries that all consumers must include to
import logic. These are cancerous and tend to make consumers builds
difficult or outright fail. All have been removed, except for the one
required by flux. This entry, while retained, has been converted into a
noop and should be removed when fluxcd/flux#2590 is fixed.
@carnott-snap
Copy link
Contributor Author

However, I am afraid that those replacements are needed. What makes you think they are not?

It may be that flux even compiles without them but that doesn't mean they are not needed.

The repo builds and tests successfully without them, and I have a basic consumer package that compiles successfully. Replace lines do not affect dependent modules (per the spec), so I am unclear why they are necessary, can you explain this requirement?

I would generally agree with that, but, they are needed. Also, could you provided an existing scenario in which the replacements are an issue?

I have interest in using transitive dependencies of flux, and do not want to maintain a list of eight replace directives for modules that I do not use (directly). There is no automated way to keep them in sync, and upgrading the flux version will cause breakage when these replace lines change.

@2opremio
Copy link
Contributor

2opremio commented Nov 7, 2019

The docker distribution replacement is needed to honor timeouts when accessing repositories. The Kubernetes replacements are used to ensure Flux backwards compatible with Kubernetes and to make sure that the go client version we use is compatible with the Kubernetes libraries. They are needed because different dependencies import different versions of the Kubernetes libraries, and unfortunately Golang doesn't support using multiple dependency versions, not letting us specify our own versions directly.

I have interest in using transitive dependencies of flux, and do not want to maintain a list of eight replace directives for modules that I do not use (directly). There is no automated way to keep them in sync, and upgrading the flux version will cause breakage when these replace lines change.

Sorry, but I don't see how that would cause things to break. As I see it, the impact of changing the replacements would not be worse than bumping the dependency versions without an explicit replacement rule.

Regardless, the replacements are there for a reason and we cannot simply remove them. We could consider bumping the replacement versions, but that's about it.

@2opremio
Copy link
Contributor

2opremio commented Nov 7, 2019

@carnott-snap to understand your usecase, why are you importing Flux as a library?

@carnott-snap
Copy link
Contributor Author

The docker distribution one is needed to honor timeouts when accessing repositories ...

I am sympathetic to this concern, however that change lives in a stale fork, and is over six months out of date. Is there work to upstream this fix? If so, we can leave the replace, but should link against the PR and warn about impending removal. If not, the logic should be internalised, so that consumers do not see the side effects and the maintenance burden is clear to the maintainers of this library.

[T]he Kubernetes ones to ensure Flux backwards compatible with Kubernetes. They are needed because different dependencies import different versions of the Kubernetes libraries, and unfortunately Golang doesn't support using multiple dependency versions.

I do not see how replace solves this problem. As I understand things, if you require a given version of a library, you get that version. I am unclear how replace gives you any more backwards compatibility than require? If I have chosen too high a version for the tag, say you want kubernetes-12.0.3, I can fix this up.

Sorry, but I don't see how that would cause things to break. As I see it, the impact of changing the replacements would not be worse than bumping the dependency versions without an explicit replacement rule.

I should be clear, while there the mentioned breakage relates to upgrades and will happen with replace or require. This is more about complexity and maintenance burden. If I have a go.mod file that works, it will continue to work. If I have two dependencies that require incompatible versions of kubernetes libraries, that will be a problem either way. My concern is that every dependency that is transitively linked to flux will need to manually keep in sync a replace block.

@carnott-snap to understand your usecase, can you elaborate on your usecase? Why are you importing flux?

We are attempting to integrate with the eksctl repo, which depends on flux: https://github.com/weaveworks/eksctl/blob/master/go.mod#L25. Are you indicating that flux is not maintained as a library? If so, can we deprecate the external api surface, so that consumers will not be confused?

carnott-snap added a commit to carnott-snap/helm-operator that referenced this pull request Nov 8, 2019
go.mod contains many replace entries that all consumers must include to
import logic. These are cancerous and tend to make consumers builds
difficult or outright fail. All unnecessary replaces have been removed.

One is required by flux, and while retained, has been converted into a
noop and should be removed when fluxcd/flux#2590 is fixed.

A co-dependant transitive upgrade was detected, that bumps
k8s.io/code-generator above kubernetes-1.14.4. To correct this the
go.mod has been left untidy, but with the correct version required,
and a replace directive has added so that build/test will pass.
carnott-snap added a commit to carnott-snap/helm-operator that referenced this pull request Nov 8, 2019
go.mod contains many replace entries that all consumers must include to
import logic. These are cancerous and tend to make consumers builds
difficult or outright fail. All unnecessary replaces have been removed.

One is required by flux, and while retained, has been converted into a
noop and should be removed when fluxcd/flux#2590 is fixed.

A co-dependant transitive upgrade was detected, that bumps
k8s.io/code-generator above kubernetes-1.14.4. To correct this the
go.mod has been left untidy, but with the correct version required,
and a replace directive has added so that build/test will pass.
carnott-snap added a commit to carnott-snap/helm-operator that referenced this pull request Nov 8, 2019
go.mod contains many replace entries that all consumers must include to
import logic. These are cancerous and tend to make consumers builds
difficult or outright fail. All unnecessary replaces have been removed.

One is required by flux, and while retained, has been converted into a
noop and should be removed when fluxcd/flux#2590 is fixed.

A co-dependant transitive upgrade was detected, that bumps
k8s.io/code-generator above kubernetes-1.14.4. To correct this the
go.mod has been left untidy, but with the correct version required,
and a replace directive has added so that build/test will pass.
@2opremio
Copy link
Contributor

2opremio commented Nov 8, 2019

I am sympathetic to this concern, however that change lives in a stale fork, and is over six months out of date. Is there work to upstream this fix?

distribution/distribution#2905 , which has been ignored by the maintainers so far.

I do not see how replace solves this problem. As I understand things, if you require a given version of a library, you get that version

Not really, in my experience Go will override the version you require by the version required by your dependencies.

My concern is that every dependency that is transitively linked to flux will need to manually keep in sync a replace block.

I don't think that's the case. You don't need to add a replace rule if you don't care about the specific version of your dependencies.

Are you indicating that flux is not maintained as a library? If so, can we deprecate the external api surface, so that consumers will not be confused?

I wasn't trying to imply that. In fact, Flux intentionally exports the install package.

@2opremio
Copy link
Contributor

2opremio commented Nov 8, 2019

If so, we can leave the replace, but should link against the PR and warn about impending removal

You are right, we should add a comment in go.mod.

@2opremio
Copy link
Contributor

2opremio commented Nov 8, 2019

I do not see how replace solves this problem. As I understand things, if you require a given version of a library, you get that version

Not really, in my experience Go will override the version you require by the version required by your dependencies.

Let's put it in a different way. If you manage to remove the replace directives without changing go.sum as a result, I will be more than happy to merge your PR.

I doubt that's possible though.

@carnott-snap
Copy link
Contributor Author

Let's put it in a different way. If you manage to remove the replace directives without changing go.sum as a result, I will be more than happy to merge your PR.

I believe 23325e1 meets these standards. There is one additional line in the go.mod, however it is only used during dependency verification and does not affect build or actual resolved versions:

k8s.io/api v0.0.0-20190313235455-40a48860b5ab/go.mod h1:iuAfoD4hCxJ8Onx9kaTIt30j7jUFS00AXQi6QMi99vA=

My initial change also included safe dependency upgrades, using @latest, this has been rolled back and I will submit these changes separately.

distribution/distribution#2905 , which has been ignored by the maintainers so far.

With the lack of traction on the PR, and the importance of this feature, it suggests we should make a real fork. I dislike imposing the replace, and think that a fork will allow you to maintain the desired logic in the long term.

This will involve publishing a valid module to github.com/2opremio/distribution (or github.com/fluxcd/distrobution, since this is the org that needs the logic) and migrate all the references from github.com/docker/distrobution.

You are right, we should add a comment in go.mod.

Since I am unsure if you are willing to take the fork route, I have amened go.mod to include this comment and am happy to revise the wording to your liking.

Not really, in my experience Go will override the version you require by the version required by your dependencies.

This is not true for the current state of the repo: a replace is required for github.com/docker/distribution because you use an non-existent pseudo-version, v0.0.0-00010101000000-000000000000. To correct this, and make replace opt in, I have replaced the pseudo-version a valid one: v2.7.1+incompatible. Unfortunately because of the dependency cycle between flux and helm-operator, this change alone is not enough. We will need to break the history before we can remove this requirement. It seems like this will benefit the helm-operator repo as well, so we can orchestrate that out of band.

go.mod Outdated Show resolved Hide resolved
@2opremio
Copy link
Contributor

This is starting to look good, thanks.

There is one additional line in the go.mod, however it is only used during dependency verification and does not affect build or actual resolved versions:

How do you know that?

@carnott-snap
Copy link
Contributor Author

carnott-snap commented Nov 12, 2019

How do you know that?

There are two types of hashes in go.sum: module and go.mod. The former is the actual version that is used for compilation, thus there is only one per module import path, where as the latter are required for every referenced version of a module in the dependency graph, thus there may be many:

k8s.io/api v0.0.0-20190313235455-40a48860b5ab/go.mod h1:iuAfoD4hCxJ8Onx9kaTIt30j7jUFS00AXQi6QMi99vA=
k8s.io/api v0.0.0-20190708174958-539a33f6e817 h1:V6YPTc5fSnwv7EBjx6es9VyAki/6bqK4M3ECA6WwfBk=
k8s.io/api v0.0.0-20190708174958-539a33f6e817/go.mod h1:iuAfoD4hCxJ8Onx9kaTIt30j7jUFS00AXQi6QMi99vA=

As such, we can read these three lines as saying: you will find two module versions, v0.0.0-20190313235455-40a48860b5ab and v0.0.0-20190708174958-539a33f6e817, then use the more recent, v0.0.0-20190708174958-539a33f6e817, to compile with.

Before you locked v0.0.0-20190708174958-539a33f6e817 with a replace to l directive, thus the toolchain did not verify any other versions of that module, and go.sum only had two lines.

The use of the correct version can also be confirmed by running go list -m k8s.io/api:

k8s.io/api v0.0.0-20190708174958-539a33f6e817

@2opremio
Copy link
Contributor

@carnott-snap makes sense, would you mind rebasing on master before I merge?

go.mod contains many replace entries that all consumers must include to
import logic. These are cancerous and tend to make consumers builds
difficult or outright fail. All have been removed, except for the one
required by helm-operator. This entry, required transitively by
helm-operator, should be removed when distribution/distribution#2905 closes.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants