Skip to content
This repository has been archived by the owner on Nov 2, 2018. It is now read-only.

Reproducible source checkouts for release builds #2388

Open
starius opened this issue Oct 2, 2017 · 32 comments
Open

Reproducible source checkouts for release builds #2388

starius opened this issue Oct 2, 2017 · 32 comments
Labels
Feature Request

Comments

@starius
Copy link
Contributor

starius commented Oct 2, 2017

As of 4fe48d5 packages under Sia/ directory depend on 84 packages, many of them are not under NebulousLabs/.

If somebody wants to build Sia 1.3.0 in a year, it will be nontrivial to checkout proper revisions of all those packages (including those under NebulousLabs). Also it is hard to sign all the source needed to build a specific version: if you sign the commit behind tag 1.3.0, most of source is not covered by this signature (because it is outside Sia/ and its changes do not invalidate the signature).

At the same time some packages are already vendored: smux, pkg/errors, twofish.

I found the tool that can fix the issue: https://github.com/kardianos/govendor.

We can remove all files currently in vendor dir and instead put vendor/vendor.json there. The vendored sources will not be used for development (instead normal Go imports will be used, as usual) and they will be added to .gitignore, but make release-std will checkout all vendored sources using govendor sync. Alternative is to put all source of dependencies to vendor - it is only 8MB - but then normal packages would not be used at all, which is likely to result in using old versions (even packages currently in vendor are not the latest revisions).
@dmmcquay
Copy link

dmmcquay commented Oct 3, 2017 •
edited
Loading

+1000

I actually brought up this idea yesterday in #core-dev but suggested a slightly different approach (different tooling).

The Go community's official tool for dependency and package management is dep and, as discussed at Gophercon 2017, should be the tool of choice going forward. Development for other tools in the same space (govendor, glide, etc.), as far as I am aware, have stopped and all the different tool communities are combining and working on dep.

As for whether or not the vendor/ directory should be tracked in vcs, there is a FAQ which talks about the pros and cons. I personally believe that anytime you import something, you are taking on the responsibility of that code, which is why I like tracking it in vcs. Also, it is the the safest way to ensure truly reproducible builds, as it guards against upstream renames, deletes, and commit history overwrites (taken from the dep FAQ). I believe that tracking it in vcs will initially also help us, as many of the packages that Sia imports are currently not tagged (which I believe is part of this issue and should be tracked along with it).

I would suggest (and would be happy to do it!) vendoring code with dep and then following up on all the packages Sia imports and make a best effort to get as many of them tagged. Obviously this includes many packages in the NebulousLabs organization, which will also help those packages and anyone who imports them.

@dmmcquay
Copy link

dmmcquay commented Oct 3, 2017

Here is my branch that I started putting together last night with what the repo would look like vendoring with dep.

Still a WIP as I am working on issue with github.com/huin/goupnp as it wasn't properly
vendor in github.com/NebulousLabs/go-upnp and no history exists about
when the snapshot of the repo was taken.

@DavidVorick
Copy link
Member

One thing I'm fairly paranoid about is retroactive malware. If you gain access to the developer's account for one of our dependencies, you could potentially sneak in malware that ends up harming our users. Vendoring is one good way to protect against this - it means that you don't update your deps until you choose, and that also means you have a chance to visually inspect all changes to make sure nothing drastic happened.

I think it's much preferable to have out of date package than to update continuously for this reason. At the same time, we do want to make sure that we're notified anytime that a dependency is updated because we need to make sure that if there's a security patch we get that asap (while still inspecting the fix for malicious intent)

@dmmcquay
Copy link

dmmcquay commented Oct 3, 2017 •
edited
Loading

@DavidVorick Exactly, using a tool like dep and bringing around the vendor/ directory in vcs protects against what you are saying.

Especially if we can get our packages that we import to tag releases, we can track this type of changes easily using dep.

Here is an example of current status of all Sia's dependencies and from which we could track changes:

PROJECT                                    CONSTRAINT     VERSION        REVISION  LATEST   PKGS USED
github.com/NebulousLabs/bolt               branch master  branch master  a22e934   a22e934  1
github.com/NebulousLabs/demotemutex        branch master  branch master  235395f   235395f  1
github.com/NebulousLabs/entropy-mnemonics  branch master  branch master  7b01a64   7b01a64  1
github.com/NebulousLabs/errors             branch master  branch master  98e1f05   98e1f05  1
github.com/NebulousLabs/fastrand           branch master  branch master  60b6156   60b6156  1
github.com/NebulousLabs/go-upnp            branch master  branch master  620e235   620e235  1
github.com/NebulousLabs/merkletree         branch master  branch master  8482d02   8482d02  1
github.com/NebulousLabs/muxado             branch master  branch master  b4de4d8   b4de4d8  5
github.com/bgentry/speakeasy               ^0.1.0         v0.1.0         4aabc24   4aabc24  1
github.com/cpuguy83/go-md2man              *              v1.0.7         1d903dc   1d903dc  1
github.com/huin/goupnp                     *              branch master  5b7801a   5b7801a  6
github.com/inconshreveable/go-update       branch master  branch master  8152e7e   8152e7e  3
github.com/inconshreveable/mousetrap       *              v1.0           76626ae   76626ae  1
github.com/julienschmidt/httprouter        ^1.1.0         v1.1           8c199fb   8c199fb  1
github.com/kardianos/osext                 branch master  branch master  ae77be6   ae77be6  1
github.com/klauspost/cpuid                 *              v1.1           ae7887d   ae7887d  1
github.com/klauspost/reedsolomon           ^1.6.0         v1.6           6bb6130   6bb6130  1
github.com/pkg/errors                      *              v0.8.0         645ef00   645ef00  1
github.com/russross/blackfriday            *              v1.5           4048872   cadec56  1
github.com/spf13/cobra                     branch master  branch master  e5f66de   0dacccf  2
github.com/spf13/pflag                     *              v1.0.0         e57e3ee   e57e3ee  1
github.com/xtaci/smux                      ^1.0.5         v1.0.5         a1a5df8   a1a5df8  1
golang.org/x/crypto                        branch master  branch master  9419663   9419663  4
golang.org/x/net                           *              branch master  0a93976   0a93976  3
golang.org/x/sys                           *              branch master  314a259   314a259  1
golang.org/x/text                          *              branch master  1cbadb4   1cbadb4  20
gopkg.in/yaml.v2                           *              branch v2      eb3733d   eb3733d  1

As you can see, NebolousLabs owned repos are a big chunk of untagged repos we import. There are some I saw that also weren't tagged, but most of the others we just have to say what release we want and then we have a much better story not only from a retroactive malware but also with reproducible builds.

@starius
Copy link
Contributor Author

starius commented Oct 3, 2017 via email

@dmmcquay
Copy link

dmmcquay commented Oct 3, 2017

@starius After running dep prune on my branch, was able to get the vendor/ dir down to ~ 16MB. Do you have a branch I can take a look at the govendor vendor/ dir? The size differences makes no sense... it is just bringing in the deps Sia uses.

@dmmcquay dmmcquay mentioned this issue Oct 3, 2017
Closed
2 tasks
@tbenz9
Copy link
Collaborator

tbenz9 commented Oct 3, 2017

It certainly seems like the Go community is gravitating towards dep as a vendoring solution and I believe Sia should begin moving towards dep as well. dep certainly protects us against 'retroactive malware' since we can still pick the tagged version for each release of Sia. And I agree with @dmmcquay I don't think a few MB make much difference in our Repo, the benefits gained from using a community supported tool like dep far outweigh an additional ~8MB (when pruned) of code that Sia requires to run.

What do we do about the Sia dependencies that are not using tagged releases?

starius pushed a commit to starius/Sia that referenced this issue Oct 3, 2017
switch to govendor
3f1a618 
fix NebulousLabs#2388
@starius
Copy link
Contributor Author

starius commented Oct 3, 2017

My branch: master...starius:vendor

@starius
Copy link
Contributor Author

starius commented Oct 3, 2017

The Go community's official tool for dependency and package management is dep

dep's homepage says "dep is the official experiment, but not yet the official tool." (emphasis by me)

It certainly seems like the Go community is gravitating towards dep as a vendoring solution

https://github.com/golang/go/wiki/PackageManagementTools refers to https://github.com/blindpirate/report-of-build-tools-for-java-and-golang#conclusion-1 for vendoring tools popularity. govendor is 3rd in the list after Makefile and godep (which has nothing to do with dep I guess). I don't see dep in the list.

@starius
Copy link
Contributor Author

starius commented Oct 3, 2017

Hmm, it reminded me the xkcd about Standards :-)

@starius
Copy link
Contributor Author

starius commented Oct 3, 2017 •
edited
Loading

There are several orthogonal issues in this issue:

  1. whether to make reliable snapshots of dependencies (I think all of us agree that yes)
  2. which tool to use (many options)
  3. whether to keep sources of deps in Sia's Git (many options)
  4. whether to use the latest version or tags of deps (many options)

@dmmcquay
Copy link

dmmcquay commented Oct 3, 2017 •
edited
Loading

It is very likely that dep will be integrated into the Go tool sometime in the future. Watching the talk from Gophercon 2017 I think will really clear up some confusion about this and what is meant by the Official tool.

As to issue brought up, I'll voice my opinion on them.

which tool to use: dep

whether to keep sources of deps in Sia's Git: deps should be brought along for the ride (that way we can ensure protection against retroactive malware by seeing what changes when we upgrade pkgs)

whether to use the latest version or tags of deps: the best scenario is to use tagged releases (see here]). If that is not possible, we have to do what is second best and do it off a commit hash. As a lot of the repos that are not tagged NebulousLabs owns, we can remedy a large majority of the problem ourselves. When I checked this morning it seemed like all of the other packages have tagged releases (have to follow up about golang.org/x/* repos, but that either is already done or in the works).

@tbenz9 as to your question, we basically tag based on a commit hash for now. Its the least safe of the options, but it is better than having everything based on a go get -u.

@lukechampine
Copy link
Member

I don't like having other projects' source code in Sia's git. It feels needless. I think my ideal dependency management tool would keep the sources outside in your GOPATH as usual, and just ensure that the correct git hash was checked out for each dependency before running go build.

That said, vendoring a copy does come with some advantages, and the disadvantages (larger repo size?) don't concern me as much as they might have in 1995. I think the best thing to do here is use whichever of dep/govendor/godep that suits our purposes best. Ideally, if one of those is eventually endorsed as the "official tool," it should be easy to switch to it.

@dmmcquay why is using a tag better than a commit hash?

@dmmcquay
Copy link

dmmcquay commented Oct 3, 2017

@lukechampine Dave Cheney says it better than I can: Gophers, please tag your releases

Whether we like the vendor/ dir or not for dependency mgmt, that is the way the world is going. go get is limited in what it can do and the vendor/ was the solution to the problem.

I don't think of it as having "other projects' source code in Sia" because the reality is that is already is the case. When you have a import anywhere, you are bringing that code and, in my opinion, you start to own it there. Where it lives I can see the argument, but, as mentioned above, the decision was already made.

Most large projects carry the vendor dir along for the ride (coming from the kubernetes community, go look at projects there and you'll see it). It really isn't an issue (just creates one large commit, but once the code is in you don't really notice.. unless you want to in the case of a pkg update which allows us to inspect the code more closely).

@starius
Copy link
Contributor Author

starius commented Oct 4, 2017

whether to keep sources of deps in Sia's Git: IMHO, no. Keep only hashes to have a buildable version and to protect against retroactive malware.
whether to use the latest version or tags of deps: IMHO, yes. "master" is the only meaningful tag in Go.

I think go get should continue working.

I don't think of it as having "other projects' source code in Sia" because the reality is that is already is the case.

  • Is it applicable to the standard library and to the toolchain? (And Go 1.4 bootstrap needed to build the toolchain?)
  • Imagine everybody uses the approach of putting all dependencies to Git. If dependencies depend themselves (package A depends on packages B and C, package B also depends on C) it is not trivial to resolve the conflicts when updating. (We can not put multiple copies of C to the binary, because then objects from C created by A's copy of C will not be compatible with B's.)

It is duty of maintainers (e.g. in Debian) not software authors to fix versions of dependencies hard. Developers normally say "A depends on B >= 1.1", then maintainers pick up particular B (for instance, 1.2) and ship it to end users. Note that this B=1.2 is global for the whole distribution (e.g. Debian): all packages depending on B use B=1.2. When we put commit hashes of dependencies to vendor/vendor.json we say "it works with this fixed versions and should also work with later versions".

Why don't "import" operators in Go contain commit hashes? I asked this question when I was new to Go. The answer is that nobody would update them: developers are lazy and maintainers don't touch things that work. Contrary, having "import"s pointing to the latest version forces people to move forward. If you clone and build a project with go get and get errors because the code was not updated for dependencies, you can throw the code away - it is not maintained. Many people use this simple go get to examine new code. Putting all dependencies to "vendor" is lying to those people.

In large companies the only version of software of interest is the latest one (in master branch). It is guaranteed to work correctly. If the package development is in progress, it can be done in a separate branch (e.g. "dev") which is merged into "master" after isolation period. This approach works really well and that is the ecosystem Go was born in and for, I guess.

@dmmcquay
Copy link

dmmcquay commented Oct 4, 2017

Keeping the code in vcs is a topic of discussion and both camps have valid points. People have different opinions, personally I like bringing it along for the ride. If we don't track the versions of the packages we depend on and only keep track of commit hashes or checksums of the code, combing through the changes will be a bear (leaving open the possibility of retroactive malware bug) . I'm open to either options and would love more to chime in to get a sort of consensus.

I disagree with you about imports in Go and I think its evident the Go team and others do as well. The reason these tools exist is to avoid building from master. The assumption that master is always clean and ready to roll, I feel, is a faulty one. This might be true in a closed environment (a company, a team, etc.) but to assume that every package we import has a CI behind it making sure that every commit to master is safe and tested isn't what I've seen. I also disagree about "it works with this fixed versions and should also work with later versions." I find that to be rather unsafe (that's why we hear stories about APIs breaking for people downstream and the misery it brings :( ...). Another issue with always living on master, or, in other words, latest is how do you roll back from latest if there is an issue with the code and it breaks something? If you have ever watched a talk by Kelsey Hightower this is something he loves to hammer home.

@starius
Copy link
Contributor Author

starius commented Oct 4, 2017

If we don't track the versions of the packages we depend on and only keep track of commit hashes or checksums of the code, combing through the changes will be a bear

Not really. You can start a branch, checkout all the packages of current commits, add whole "vendor" dir to Git, update packages to the latest version and run git diff. The same as having them in Git but only for update-screening purposes.

I disagree with you about imports in Go and I think its evident the Go team and others do as well.

Was there a message from the Go team discouraging usage of traditional imports?

to assume that every package we import has a CI behind it making sure that every commit to master is safe and tested isn't what I've seen

I believe it is true for majority of Go packages. If we find ourselves using a package that has broken master, we can either fork working commit of it to NebulousLabs/ or stop using the package.

When I build software (not in Go) having copies of other software source in its tree the first thing I do is removing those copies and patching it to use copies provided by my environment.

@lukechampine
Copy link
Member

You can also easily compare commits using GitHub, e.g. NebulousLabs/bolt@657f184...a22e934

I agree with @dmmcquay that we shouldn't be building from master. I thought that was kind of the point, honestly. It's not that we're worried about a dependency changing their APIs and breaking our build, it's that we're worried about someone pushing malicious code to a dependency's master. That's why we want to specify the exact commit hash. We'd probably want to do this even for Nebulous-controlled repos, just in case one of them gets compromised.
(As a side note, another problem with building from master is that it makes atomic dependency updates impossible, at least in theory. e.g. if we want to make a breaking API change to our bolt fork, then there's no way to avoid temporarily breaking the Sia build, since it will be out-of-sync with the API on the bolt master until the new commits have been pushed to both repos. If we specify a commit hash, we can push the bolt commit first, then atomically update both the commit hash and the API usage in a single Sia commit.)

As for why go get only supports fetching the latest master, I think it's likely a consequence of Go having been built at Google, where it's both safe and necessary to keep all your dependencies up to date. See https://groups.google.com/forum/#!topic/golang-dev/nMWoEAG55v8%5B1-25%5D for a discussion of how Google handles vendoring (well, how they handled it in 2015, at least).

@starius, there's also some discussion about how best to manage modifying our dependencies (here). One of the obvious disadvantages of not copying the sources is that we can't easily modify them as needed. We've already done this at least once (for our go-upnp package). If we're not copying the sources, we need to either get our changes merged upstream, or fork the repo. Not saying that these are necessarily bad (some would argue the opposite, i.e. that vendoring discourages open source contributions), just something to be aware of.

@DavidVorick
Copy link
Member

Tags are not secure because you can change what commit hash your tag refers to. Therefore it's not useful if the concern is stopping retroactive malware.

Having a commit hash is also not good enough because there's no guarantee that you'll be able to find code that maps to that commit hash. We need to make sure that we preserve that code somewhere, and if that's not in the Sia repo itself, it needs to be in an NL repo at the very least. Otherwise we are vulnerable to data witholding-classes of attack

@starius
Copy link
Contributor Author

starius commented Oct 4, 2017

TL;DR: use latest version of dependencies for development and fixed for releases.

we shouldn't be building from master

We should not build production binaries (included in releases) from dep's master. But we should use dep's master for development: to keep up with new bugs asap, not when releasing.

we're worried about someone pushing malicious code to a dependency's master.

We'll have more chances to discover it (and not putting into release binary) if we use dep's master during development.

if we want to make a breaking API change to our bolt fork, then there's no way to avoid temporarily breaking the Sia build

It is possible to push such change in 3 steps (make backward compatible change in the dep first, then update main repo to use new API, then remove old API from dep). It should not be a common case though.

where it's both safe and necessary to keep all your dependencies up to date

Keeping dependencies up to date in development environment is both safe and necessary in open source as well. Infecting the development environment can't be avoided, because you have to update dependencies at some point. Infecting development environment shouldn't result in a disaster anyway. What we can (and must) do is avoiding infecting released binaries - by discovering bad commit in a dependency soon.

we need to either get our changes merged upstream, or fork the repo

Or vendor and fix there. IMHO, the best approach is fix locally (in vendor dir or in fork) and keep there until it is merged upstream. The "vendor" dir doesn't bring much in this situation, if you track source and commit of dependencies in your repo.

We need to make sure that we preserve that code somewhere

I think the safest place is our local hard disks where it is already :-)
In theory NL fork can be also withhold (e.g. by DMCA request).

@avahowell
Copy link
Contributor

Perhaps another reason to fully vendor our dependencies: go get has attack surface as revealed in the recent security patch to Go (https://groups.google.com/forum/m/#!topic/golang-nuts/sHfMg4gZNps). I think I'm in favor of fully vendoring our dependencies, but if we do that we also need to make an effort monitor and update them if/when critical security fixes arise. In Sia-UI we use a package-lock.json to freeze the entire dependency tree, and we continuously scan that tree for critical fixes.

As for pulling from master in development and using fixed versions for release, I would be concerned there about subtle divergent behavior. For example, it's plausible that our test suite could pass when using latest dependencies, but fail with our pinned production dependencies. Also, infecting a developer's environment with retroactive malware is still quite awful. Not sure I understand @starius 's point about it being unavoidable because you have to update dependencies at some point. If we control and review every update made to our dependencies it seems far less likely that malware could slip in.

@starius
Copy link
Contributor Author

starius commented Oct 6, 2017

I didn't know about the go get bug - thanks for that!

What do you think about vendoring the standard library and the toolchain? Few weeks ago I spend hours figuring out how to clean one project which has unusual source control system and build system, including several gigs of gcc binaries for several targets in the repo. I wanted to make sure I didn't run any of them and finally wrote a script removing everything but sources before starting the build. But I still don't know exactly how that thing builds. I wish that project was written in Go and build with just go get.

The priorities for build for dev and for prod are different. The former is interested in fast build using the latest (or even locally modified) version of dependencies (exactly what go get does). The later is interested in security and reproducibility of source and binaries. It is hard to coexist for these two opposing goals in one place so in most cases they don't: devs do their job in one place (e.g. Github) and maintainers do their job in a separate place (e.g. Debian) taking snapshots of the development version from time to time. However I believe that for a Go package it is still possible to do it in one place, just interests of both developers and maintainers should be represented (i.e. ability to do both fast & dirt builds with go get and secure builds for releases).

I was not accurate saying that the risk of infecting the dev environment is unavoidable - it is hard to avoid, but still possible. But instead of securing dev environment you can give developers maximum freedom (including root on the machine to install anything) but no privileges (like git push) and only secure the way you copy the changes to the clean environment. This approach can be implemented using a separate (virtual) machine for development. Changes can be transferred using Git patch files. This approach would prevent any damage of the bug in go get - because go get would run only in an environment having no privileges.

it's plausible that our test suite could pass when using latest dependencies

We can run two Travis jobs: one for dev and another for prod. The former builds just with go get and the later pulls fixed commits of dependencies and then builds.

@tbenz9
Copy link
Collaborator

tbenz9 commented Oct 6, 2017

What do you think about vendoring the standard library and the toolchain?

Yeah, I don't think that will be necessary. Seems extremely excessive to me.

The priorities for build for dev and for prod are different.

I can see your point, but I don't think the priorities are that different. What is dev hopefully becomes prod eventually. I think they should be treated as similarly as possible (in general).

My 2 cents: Use dep, bring in the dependencies, use hashes to mark our dependencies, tag our releases.

@lukechampine
Copy link
Member

What do you think about vendoring the standard library and the toolchain?

This would likely be necessary in order to create bit-perfect deterministic builds. However, the current Go toolchain doesn't appear to support deterministic compilation anyway, so it's out of scope for now.

I think the best we can do here is:

  1. Always use tagged Go releases when building production binaries
  2. Specify (in the release notes) which Go version was used to compile the binaries

That way we can at least identify and reproduce any bugs or security issues introduced by a specific Go toolchain.

@lukechampine
Copy link
Member

I think at this point we've agreed that we must fully vendor all our dependencies. As per @DavidVorick's comment, we should also be pinning those dependencies to commit hashes, not tags (although IIUC this is less important when the sources are fully vendored). That just leaves the decision of which vendoring tool to use. dep seems like the obvious choice, but one of the alternatives might fit our needs better.

@starius starius mentioned this issue Oct 7, 2017
Open
@starius
Copy link
Contributor Author

starius commented Oct 7, 2017

bit-perfect deterministic builds

I think we should provide a shell script producing the released binary from scratch: downloads and installs Go, downloads Sia and all its dependencies and buildes them. Then everybody can run the script and compare results bit-by-bit. See also #2410

Let me summarize arguments.

Pro-fully vendoring (put all sources of deps to vendor dir):

  • If a dependency's repo is removed our build is broken

Pro-hash-only vendoring (putting only hashes/commits of dependencies to Sia source):

  • Developers have choice against which dependencies to build: vendored or local
  • Several times smaller source and Git size. Smaller size of diffs in git log -p

Both decisions provide:

  • Source reproducibility (if a repo is removed, we always have hash to verify somebody's saved copy)
  • Protection against vulnerabilities in go get when building for prod
  • Ability to write a script for bit-perfect deterministic build (Write a script for bit-perfect deterministic build #2410)

@starius
Copy link
Contributor Author

starius commented Oct 8, 2017

I tried to write a script for #2410 and discovered few things which made me less enthusiastic about the hash-only approach:

  • fully vendoring has another benefit: the only build dep is Go. In hash-only approach we have a chicken-and-egg problem: the vendoring tool (dep or govendor) itself has multiple dependencies which have to be downloaded at specific commits.
  • both dep or govendor use sha1 not sha256. Sha1 is considered weak now. Git commits are sha1 checksums. In addition, govendor provides sha1 checksums (of trees I guess) encoded in base64. dep provides only commit hashes. This is another argument for fully vendoring (unless we find a vendoring tool using strong checksums).

Can dep write sha256 to lock files?

@dmmcquay
Copy link

I think if we fully vendor and when releasing versions of Sia post what the hash of the binary needs to be, we can eliminate a portion of the security concern with bit-perfect builds, etc.. This puts the responsibility on us when releasing versions but this might be a path to quell these concerns a bit.

Since we seem to be leaning towards committing the vendor/ dir, questions about sha1 vs sha256 or using git commit hashes vs tags becomes less relevant. They then just become tools that allow us, when upgrading, to better understand what is happening (example is going from version 1.0 to 1.1 rather than just commit hash to other commit hash) but still put the responsibility of seeing what changed in the code (which would be possible because it is tracked in vcs).

I would propose taking a look at this PR and adding comments there. It seems to me that dep also seems to be the tool most agree on should be used.

@starius
Copy link
Contributor Author

starius commented Oct 12, 2017

I changed my mind and now lean towards committing the vendor.
The reason: to avoid the chicken-and-egg problem in the script building reproducible binaries.

The PR looks good to me and I used it in my script building reproducible binaries. One comment: it bundles tests and other unneeded files. Could you vendor only Go files that are actually needed + README, please?

@lukechampine
Copy link
Member

We would also need the LICENSE files, right?

Personally I think it's fine to include tests, but if there's a dependency that includes some 5MB testdata file or something, we should eliminate that

@dmmcquay
Copy link

These are things handled by dep, it’s not something you mess with manually otherwise you’ll have to keep track of each of these changes and would have no tooling around rebuilding those changes. iirc, If you run ‘dep prune’ it should remove many of these concerns.

@starius
Copy link
Contributor Author

starius commented Oct 21, 2017

add "Feature Request" Label

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Feature Request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

vendor: adding dependency mgmt with dep dmmcquay/Sia
6 participants
@starius @dmmcquay @tbenz9 @DavidVorick @lukechampine @avahowell