Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

keep old packages #56

Closed
MWigger opened this issue Jun 13, 2017 · 12 comments
Closed

keep old packages #56

MWigger opened this issue Jun 13, 2017 · 12 comments
Labels
bug

Comments

@MWigger
Copy link

MWigger commented Jun 13, 2017

Hello,
I have a little struggle with conda-mirror:
We use environments where we sometimes define package versions.
Unfortunately conda-mirror deleted one of them, since it was "to old"
Is there a way to keep old packages, or even better: have a blacklist of packages not to remove?
@ericdill
Copy link
Contributor

Hi @MWigger. Thanks for the issue report. What version of conda-mirror are you running?

Unfortunately conda-mirror deleted one of them, since it was "to old"

I suspect what happened was that conda-mirror deleted a package that you had locally because it was removed from the upstream repository. There is currently no mechanism that checks the age of package. There are four conditions where a package will be removed.

  1. Package was deleted from the upstream repository
  2. Package is an invalid tarball
  3. Package fails the md5 checksum
  4. Package fails the sha256 checksum

If you were mirroring the defaults channel or the conda-forge channel, I can check my own logs at that time to see if I can help debug this issue.

Can you let me know which package was removed and the date that you noticed it? If you're not comfortable posting that information to github, please feel free to send me an email at [email protected] and I can help you debug this in a 1:1 conversation.

@MWigger
Copy link
Author

MWigger commented Jun 13, 2017 via email •
edited by ericdill
Loading

@ericdill
Copy link
Contributor

Are you logging the output of conda-mirror? Whenever it removes packages it notes why. What is the upstream channel that you are mirroring?

I see these errors in my logs for the anaconda channel:

WARNING: Removing: /opt/maxpoint/data/conda/www/upstream-anaconda/osx-64/mkl-2017.0.1-0.tar.bz2. Reason: Failed size test
...
WARNING: Removing: /opt/maxpoint/data/conda/www/upstream-anaconda/win-64/mkl-2017.0.1-0.tar.bz2. Reason: Failed size test
...

FWIW conda-mirror will attempt to redownload packages every time you run it. You should be able to run conda-mirror again and it will attempt to download again any packages that failed validation the last time you ran it.

@ericdill
Copy link
Contributor

What I neglected to add to the last comment is that I currently have mkl-2017.0.1-0 locally. It did fail once, but has not failed in the last 3 or so weeks.

@MWigger
Copy link
Author

MWigger commented Jun 13, 2017 via email •
edited by ericdill
Loading

@ericdill
Copy link
Contributor

Yes. Any package removal gets dumped as warning level which is -v

@MWigger
Copy link
Author

MWigger commented Jun 13, 2017 via email •
edited by ericdill
Loading

@MWigger
Copy link
Author

MWigger commented Jun 14, 2017 via email •
edited by ericdill
Loading

@ericdill
Copy link
Contributor

Thanks for the updated information @MWigger .

There are two distinct pieces happening there.

The first is the expected behavior of conda-mirror removing packages locally that have been removed upstream. The intent of conda-mirror is to be a complete mimic of what is available upstream. It is unfortunate that upstream removed a package that you were depending on locally. That being said, when packages are removed from upstream there is usually a good reason for this. I have zero control over or insight into the reasons for package removal from upstream.

The EOFError is something that I need to be catching. That's a new bug. Thanks for finding that 😀

@ericdill ericdill added the bug label Jun 14, 2017
@MWigger
Copy link
Author

MWigger commented Jun 15, 2017

Well, I just saw thast for example for the cffi package there is a 1.7.0 version on the coda-forge channel: https://anaconda.org/conda-forge/cffi/files

So it might be a different issue

@ericdill
Copy link
Contributor

I have seen similar package removals for 'failed size test'. I dug in to a couple of these occurances and what I found was that the package, let's take cffi-1.7.0-py27_0.tar.bz2 for example, had been deleted from the repo that I was mirroring and re-uploaded with one or more pieces of metadata changed. I think one time I looked, a license file had been added to the package. Since this does not affect the run time behavior of the package, bumping the build number (to cffi-1.7.0-py27_1.tar.bz2) was likely deemed unnecessary. Indeed only people that are maintaining their own local copy of the entire upstream repository would notice.

The reason why re-uploading a package with different metadata will cause conda-mirror to fail the size test is that the new uploaded package on upstream has a different size. The repodata.json file for that upstream repo has already been updated for the packages new size so when I compare the expected size in the upstream repodata.json with the size of the package that has been mirrored locally, they do not match. Because I treat the upstream repodata.json as the source of truth, that means the local package is now "incorrect" and it get's removed. Then the package that was just removed because of its "incorrect" size will be redownloaded from upstream.

@ericdill ericdill mentioned this issue Aug 7, 2017
Merged
@ericdill
Copy link
Contributor

ericdill commented Aug 7, 2017

Closed by #58

@ericdill ericdill closed this as completed Aug 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ericdill @MWigger