Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pkg 6001 win cf #53

Open
wants to merge 39 commits into
base: master
Choose a base branch
from
Open

Pkg 6001 win cf #53

wants to merge 39 commits into from

Conversation

ELundby45
Copy link

@ELundby45 ELundby45 commented Jan 16, 2025

Changes

  • Remove windows skips
  • Unvendor protobuf and sleef
  • Relax .* pins that (for reasons I don't understand) cause inability to create the environment on windows
  • Fixes a few tests
  • Set PYTORCH_BUILD_NUMBER=0 to prevent pytorch version from being incorrect.
  • Don't use BUILD_LIBTORCH_WHL and BUILD_PYTHON_ONLY flags anymore
    • Using these flags caused several issues including shm.dll not getting built in libtorch
  • Bring in changes from conda-forge
  • Adds upstream .gitignore in order to be able to run setup.py clean
  • Run a clean in-between pytorch builds. Fixes an intermittent problem when compiling with 3.12
    • ~50% of the time, 3.12 wouldn't get fully rebuilt resulting in LINK : fatal error LNK1104: cannot open file 'python311.lib'

Notes

  • There are several unit tests failing, but this has been the case for the last several releases.

@anaconda-pkg-build
Copy link

Linter check found the following problems: WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps

ERROR conda.cli.main_run:execute(125): conda run conda-lint /tmp/abs_10ne2dgnfa/clone failed. (See above for error)
The following problems have been found:

===== WARNINGS =====

  • clone/recipe/meta.yaml:123: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:144: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:327: python_build_tool_in_run: output "pytorch": The python build tool setuptools is in run depends
  • clone/recipe/meta.yaml:269: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:0: no_global_test: Global tests are ignored in multi-output recipes.
  • clone/recipe/meta.yaml:268: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:149: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:284: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:126: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:286: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:125: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:267: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:292: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:124: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:266: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:141: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:287: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:143: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.

===== ERRORS =====

  • clone/recipe/meta.yaml:1: remove_python_pinning: On arch specific packages, python deps should not be constrained.
  • clone/recipe/meta.yaml:187: outputs_not_unique: output "libtorch": Output name is not unique
  • clone/recipe/meta.yaml:292: remove_python_pinning: On arch specific packages, python deps should not be constrained.
  • clone/recipe/meta.yaml:0: missing_section: output "libtorch": The requirements section is missing.
  • clone/recipe/meta.yaml:53: missing_hash: The recipe is missing a sha256 checksum for a source file
    ===== Final Report: =====
    5 Errors and 18 Warnings were found

@anaconda-pkg-build
Copy link

Linter check found the following problems: WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps WARNING:root:Failed to render using RecipeParserDeps

ERROR conda.cli.main_run:execute(125): conda run conda-lint /tmp/abs_e3cm9g17iz/clone failed. (See above for error)
The following problems have been found:

===== WARNINGS =====

  • clone/recipe/meta.yaml:270: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:142: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:285: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:144: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:269: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:287: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:0: no_global_test: Global tests are ignored in multi-output recipes.
  • clone/recipe/meta.yaml:145: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:268: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:125: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:293: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:124: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:288: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:328: python_build_tool_in_run: output "pytorch": The python build tool setuptools is in run depends
  • clone/recipe/meta.yaml:127: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:150: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:267: host_section_needs_exact_pinnings: output "pytorch": Linked libraries host should have exact version pinnings.
  • clone/recipe/meta.yaml:126: host_section_needs_exact_pinnings: Linked libraries host should have exact version pinnings.

===== ERRORS =====

  • clone/recipe/meta.yaml:0: missing_section: output "libtorch": The requirements section is missing.
  • clone/recipe/meta.yaml:53: missing_hash: The recipe is missing a sha256 checksum for a source file
  • clone/recipe/meta.yaml:188: outputs_not_unique: output "libtorch": Output name is not unique
  • clone/recipe/meta.yaml:54: missing_hash: The recipe is missing a sha256 checksum for a source file
  • clone/recipe/meta.yaml:1: remove_python_pinning: On arch specific packages, python deps should not be constrained.
  • clone/recipe/meta.yaml:293: remove_python_pinning: On arch specific packages, python deps should not be constrained.
    ===== Final Report: =====
    6 Errors and 18 Warnings were found

@ELundby45 ELundby45 marked this pull request as ready for review January 21, 2025 16:03
@ELundby45 ELundby45 changed the title Pkg 6001 win cf Skip CI Pkg 6001 win cf Jan 21, 2025
@ELundby45 ELundby45 changed the base branch from PKG-6001-win to master January 21, 2025 21:25
@ELundby45 ELundby45 changed the title Skip CI Pkg 6001 win cf Pkg 6001 win cf Jan 21, 2025
@ELundby45 ELundby45 requested a review from danpetry January 21, 2025 21:31
Copy link
Contributor

@danpetry danpetry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the logs, the whole thing is building four times. Well, if it works, probably time to move on and pull conda-forge for the next version. I also checked the package contents thoroughly against conda-forge's (listed files and did a diff for all subpackages) and they are exactly identical. (apart from pytorch-cpu, which I didn't compare because it's different for us).
Testing wise I think it's ok.. more failures than I remember for test_ops so might be worth running a build of the last version and seeing how it compares, maybe. (you could only build one python version and omit other tests to speed it up.)

also is it possible/feasible to squash changes you made compared with conda-forge into one commit so we can easily apply them later, once we've synchronised our recipe? (process being 1. do a hard sync with conda-forge 2. re-apply our bespoke changes on top 3. debug)

.gitattributes Show resolved Hide resolved
abs.yaml Outdated Show resolved Hide resolved
recipe/bld.bat Show resolved Hide resolved
recipe/bld.bat Show resolved Hide resolved
recipe/meta.yaml Outdated Show resolved Hide resolved
recipe/meta.yaml Outdated Show resolved Hide resolved
recipe/bld.bat Show resolved Hide resolved
recipe/meta.yaml Show resolved Hide resolved
@ELundby45 ELundby45 changed the title Pkg 6001 win cf CI Skip Pkg 6001 win cf Jan 23, 2025
@ELundby45 ELundby45 changed the title CI Skip Pkg 6001 win cf Pkg 6001 win cf Jan 23, 2025
@h-vetinari
Copy link
Contributor

Note that we have a pretty big raft of changes to the initial windows roll-out lined up in conda-forge/pytorch-cpu-feedstock#328. I'm also looking forward to getting Anaconda's pybind-related changes through conda-forge/pytorch-cpu-feedstock#318 soon.

Sorry that this is a big mess for synchronization at the moment - there's been a flurry of changes to catch up with the former pytorch channel, and especially with getting windows in shape...

Copy link
Contributor

@danpetry danpetry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy here, thanks!

@danpetry
Copy link
Contributor

NB that you can take builds from the staging channel and copy them to zeus (for now). The validity of this approach being dependent on whether any significant changes were made after they were created.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants