Python 3.9.0rc recipe #389

ocefpaf · 2020-09-16T19:20:10Z

xref #270. @willingc You opened it but we never addressed this properly. The main issue is bandwidth to catch up with upstream, even if that will happen only on the RCs. I'm investigating the patches here and I hope to be able to work with upstream to see if we can get some of them merged (or dropped for something better).

…nda-forge-pinning 2020.09.16.18.10.04

conda-forge-linter · 2020-09-16T19:20:16Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

ocefpaf · 2020-09-16T20:00:19Z

recipe/meta.yaml

+      - patches/0012-Fix-find_library-so-that-it-looks-in-sys.prefix-lib-.patch
+      - patches/0013-Disable-new-dtags-in-unixccompiler.py.patch
+      - patches/0014-Fix-cross-compilation-on-Debian-based-distros.patch
+      # - patches/0015-Disable-registry-lookup-unless-CONDA_PY_ALLOW_REG_PA.patch


@mingwandroid this is inside a if (!Py_IgnoreEnvironmentFlag) now. Does that solve our problem or do we still need to refresh this patch.

I highly recommend keeping this. Without it, installers will fail, for example. We don't set Py_IgnoreEnvironmentFlag and until we do (if we want to) and get this enforced in all the places we need to this is crucial.

Actually I am not sure. I mean, this is our own var that some may have used.. if no one is using it then yeah, let's drop it.

willingc · 2020-09-16T22:17:44Z

@ocefpaf No worries; we're all busy. Thanks for looking into it. If not this release, there will always be a new one 😉

ocefpaf · 2020-09-19T00:51:34Z

@mingwandroid all the patches are refreshed. It looks like we can drop 0021-roll-back-2e33ecd-for-py_compile.main.patch and I'm not 100% sure I got =0015-Disable-registry-lookup-unless-CONDA_PY_ALLOW_REG_PA.patch correctly, but the rest seems fine.

Do you have an idea of which ones we should prioritize and try to push upstream?

I'll investigate the Windows failure later, not sure what is going on. Linux and macOS are building correctly, the problem seems to be the lack of python_abi metapackage for Python 3.9. (Thanks @isuruf for pointing that out to me.)

mingwandroid · 2020-09-19T08:57:27Z

Hey @ocefpaf,

My thoughts on upstreaming/dropping, patch-by-patch..

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0000-Fix-off-by-one-error-in-_winapi_WaitForMultipleObjec.patch
.. is in review upstream already. There was some pings about my PR yesterday. We can drop this once merged. Without it, you cannot install Miniconda or Anaconda (or do various other things) on a 64+ thread Windows machine. John Carmack raised this bug with us on twitter!

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0001-Add-Anaconda-Distribution-version-logic.patch
Not appropriate for upstream unless someone fancies tackling a general way to hook this bannering stuff. Anyway, conda-forge do not apply this one AFAIR.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0002-Darwin-Add-arch-x86_64-to-UnicCCompiler-ld_args.patch
Probably needs fixing for osx-aarch64 (ping @isuruf) maybe it can be removed? Worth checking.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0003-Win32-Change-FD_SETSIZE-from-512-to-2048.patch
Should be pushed upstream so that upstream Python is similarly capable.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0004-Win32-distutils-Add-support-to-cygwinccompiler-for-V.patch
Good simple bug fix, may want to add vs2019 stuff to it and push it upstream.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0005-Do-not-pass-g-to-GCC-when-not-Py_DEBUG.patch
Not sure about this one. We may want to check the size of extension modules if we removed it and if there is a difference then think about an upstreaming strategy for it.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0006-Support-cross-compiling-byte-code.patch
This has been important when needing to bootstrap our Python on a different linux arch. I don't know if things are better upstream in this regard but since this patch rebases cleanly my firm suspicion is no, it hasn't. I expect the chances of upstream wanting to touch this sort of code to be low.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0007-Win32-Fixes-for-Windows-GCC-interop-needed-by-RPy2-a.patch
Good, simple fix, should be upstreamed IMHO.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0008-Darwin-Look-in-sysroot-usr-lib-include-if-sysroot-is.patch
We could try to build without this, if it works (for building a load of extensions without problem) then consider dropping it, otherwise we should think about upstreaming it.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0009-runtime_library_dir_option-Use-1st-word-of-CC-as-com.patch
This patch allows setting e.g. "CC=gcc -g0", or "CC=ccache gcc", we should perhaps try to upstream it and get some opinions on it. I wonder if for correctness, we should be splitting by IFS though instead of ' '.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0010-Win32-Do-not-download-externals.patch
Not appropriate for upstreaming, this is to speed up our builds by not downloading things we don't need.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0011-Add-support-for-_CONDA_PYTHON_SYSCONFIGDATA_NAME-if-.patch
Not appropriate for upstreaming unless someone wants to have a go at supporting this sort of thing upstream in a way that they're likely to approve of. I doubt they would want this though. It's just a necessary evil for us, IMHO.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0012-Fix-find_library-so-that-it-looks-in-sys.prefix-lib-.patch
This patch makes Python better respect HFS (executable_path/../lib) when looking for dylibs on macOS. We could propose it upstream but it is quite conda-ecosystem specific IMHO.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0013-Disable-new-dtags-in-unixccompiler.py.patch
Here, conda-build is opinionated (as am I) about the new-dtags feature of glibc/the GNU linker. This prevents DSO interposition on Linux which many consider a security risk. The comment gives more detail. I don't know how upstream would feel about it, probably some would like it and some would dislike it!

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0014-Fix-cross-compilation-on-Debian-based-distros.patch
This should be pushed upstream. Debian-ism paths shouldn't be looked in when cross compiling.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0015-Disable-registry-lookup-unless-CONDA_PY_ALLOW_REG_PA.patch
This is important and not upstreamable IMHO.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0016-Unvendor-openssl.patch
https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0017-Unvendor-sqlite3.patch
Conda-ism, this implements our preference to use our own shared libraries. Not appropriate for upstream.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0018-venv-Revert-a-change-from-https-jackfan.us.kg-python-cp.patch
Conda-isms. If we don't want to do this then we'd need to align our layout on Windows better with upstream. I expect that would break a lot of things.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0019-Win-Add-back-the-DLLSuffix-used-to-find-the-openssl-.patch
@mariusvniekerk? Looks like something upstream might consider. But we'd likely need to also change the OpenSLL vcxproject files (that we do not use but upstream does, I think!)

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0020-Use-ranlib-from-env-if-env-variable-is-set.patch
We should try to get this upstreamed. I don't think it's very controversial (though changing the build system of Python in any way is somewhat!)

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0021-roll-back-2e33ecd-for-py_compile.main.patch
@duncanmmacleod? Seems like a bug upstream to me and we could try to upstream this fix.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0022-Add-CondaEcosystemModifyDllSearchPath.patch
I would rather not have had to write any of this but changes to upstream made it necessary. If you don't use this you will have problems, or if you don't your friends on Windows will. For my perspective 0% chance of being upstreamed.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0023-Add-d1trimfile-SRC_DIR-to-make-pdbs-more-relocatable.patch
Upstream would likely consider this.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0024-Doing-d1trimfile.patch
.. and maybe (along with) this.

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0025-egg-debugging-with-Windows-debug-builds.patch
There's a bug in egg stuff on Windows, but no one ever uses it. This was something I added when investigating that but I've completely lost the thread of that work now! Drop?

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0026-Revert-part-of-https-bugs.python.org-issue33895-http.patch
@isuruf and I are in alignment that we'll neuter this patch but make it something you can use via an env var to aid debugging.

So we need a plan of attack. I'm more than happy to ask for time to be set aside for me to handle pushing these, starting with the easy ones first and seeing how far I go, but anyone is (always) welcome to propose any of my patches and ping me in the comments if they wish.

mingwandroid · 2020-09-19T08:58:35Z

I used AnacondaRecipes URLs in my last comment, sorry about that, but it's to make it easier for me when/if I tackle this.

duncanmmacleod · 2020-09-21T08:43:17Z

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/patches/0021-roll-back-2e33ecd-for-py_compile.main.patch
@duncanmmacleod? Seems like a bug upstream to me and we could try to upstream this fix.

@mingwandroid, this is already fixed upstream, see bug report and fix python/cpython#17134. Actually, that specific patch isn't in 3.9.0rc1, but I've checked the contents of that tarball and the bug that my patch fixed isn't there, so this one can be dropped either way.

ocefpaf · 2020-09-21T11:39:13Z

@mingwandroid thnaks for the compilation of the patch status. @duncanmmacleod indeed the 0021-roll-back-2e33ecd-for-py_compile.main.patch seems to be in, I commented it out and I'm planning of removing it next iteration.

I'm copying #389 (comment) in a doc so we can tackle one at a time. We already have @willingc 👀 here ;-p

mingwandroid · 2020-09-21T12:47:41Z

@zooba may want to take a look too?

isuruf · 2020-09-24T22:31:21Z

The test error is because of a cython dependency in tests. not sure why it was added. Removing cython should unblock the tests.

recipe/meta.yaml

isuruf · 2020-09-25T14:00:08Z

recipe/meta.yaml

+        - ripgrep
+        - cmake
+        - ninja
+        - setuptools


@ocefpaf, need to remove this too

isuruf · 2020-09-25T14:00:30Z

recipe/meta.yaml

+        -   pushd distutils.cext
+        -     python setup.py install -v -v
+        -     python -c "import greet; greet.greet('Python user')" | rg "Hello Python"
+        -   popd


These 4 lines too

mingwandroid · 2020-09-25T14:07:26Z

Rather than removing all these tests, can you elide them when the patch number is 0? I think that way, we can do a build 1 once the necessary deps are present and get the benefits of the tests back.

conda-forge-linter · 2020-10-05T21:15:16Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

The recipe must have a build/number section.
The outputs section contained an unexpected subsection name. string is not a valid subsection name.

conda-forge-linter · 2020-10-05T21:16:31Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

The recipe must have a build/number section.

mingwandroid · 2020-10-06T06:51:09Z

This builds now. What's the plan with tzdata though?

mingwandroid · 2020-10-06T07:08:20Z

Does anyone know why the linter is complaining? Each output has build/number: 0 now.

conda-forge-linter · 2020-10-06T14:16:48Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

mingwandroid · 2020-10-06T15:48:13Z

This doesn't address the tzdata thing. We can do that in a follow up PR?

ocefpaf · 2020-10-06T16:33:34Z

@mingwandroid are you OK pointing this to the rc label, switching to the rc2, and merging it there first? That way we can wait for the tzdata stuff and prepare this for master in another PR.

ocefpaf · 2020-10-06T16:43:19Z

This builds now. What's the plan with tzdata though?

#392

@mingwandroid awesome work! We will merge this in the rc label for now, without the tzdata, and I'll port this recipe to master as soon as we have conda-forge/tzdata-feedstock#1

github-actions · 2020-10-06T17:26:07Z

Hi! This is the friendly conda-forge automerge bot!

I considered the following status checks when analyzing this PR:

linter: passed
azure: passed

Thus the PR was passing and merged! Have a great day!

mingwandroid · 2020-10-06T21:14:43Z

Nice. Thanks

ocefpaf added 3 commits September 16, 2020 16:12

sync with master

f46b8b4

update to 3.9 and comment out patches that do not apply

e9a2625

MNT: Re-rendered with conda-build 3.20.2, conda-smithy 3.7.10, and co…

6220041

…nda-forge-pinning 2020.09.16.18.10.04

ocefpaf commented Sep 16, 2020

View reviewed changes

ocefpaf force-pushed the 3.9rc1 branch 2 times, most recently from 2888305 to 6220041 Compare September 17, 2020 15:53

refresh patches and update to rc2

909c6df

ocefpaf force-pushed the 3.9rc1 branch from 4e48b7c to 909c6df Compare September 18, 2020 18:49

ocefpaf changed the title ~~Python 3.9.0rc1 recipe~~ Python 3.9.0rc recipe Sep 18, 2020

ocefpaf added 2 commits September 18, 2020 16:10

better comments

f978d9a

ignore_run_exports for libpython-static

8c296b4

ocefpaf force-pushed the 3.9rc1 branch from 195fed5 to 8c296b4 Compare September 18, 2020 22:58

Fix ignore_run_exports

b203ee2

xhochy mentioned this pull request Sep 24, 2020

Fix off-by-one on Windows, sync patch from Anaconda #391

Merged

5 tasks

add off by one patch

0f1f15b

ocefpaf force-pushed the 3.9rc1 branch from c5b4926 to d1aa46e Compare September 24, 2020 23:27

ocefpaf commented Sep 25, 2020

View reviewed changes

recipe/meta.yaml Outdated Show resolved Hide resolved

isuruf reviewed Sep 25, 2020

View reviewed changes

recipe/meta.yaml Outdated

- ripgrep

- cmake

- ninja

- setuptools

Copy link

Member

isuruf Sep 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ocefpaf, need to remove this too

isuruf reviewed Sep 25, 2020

View reviewed changes

ocefpaf force-pushed the 3.9rc1 branch from e4b0b77 to 154e544 Compare September 25, 2020 20:49

mingwandroid added 6 commits October 3, 2020 03:25

Oops

71a5cbf

posix package is not needed at build time on Windows

65b0cf9

Odd, add a missing_dso entry for a UCRT lib?

87ff827

Fix script (shouldn't be build/script)

07cd026

Need ncurses run dep due to lack of run_exports in CF

05da4e3

3.9.0 is released now

ee36042

Oops

0038b09

mingwandroid added 3 commits October 6, 2020 10:50

Placate the linter

87c9d26

Add top-level build deps of patch-y software

266a4b6

Por favor linter

e5c9245

isuruf changed the base branch from prerelease to master October 6, 2020 15:43

isuruf requested review from jakirkham, jjhelmus, msarahan, pelson and scopatz as code owners October 6, 2020 15:43

isuruf changed the base branch from master to prerelease October 6, 2020 16:39

ocefpaf added the automerge Merge the PR when CI passes label Oct 6, 2020

github-actions bot merged commit bdf1d97 into conda-forge:prerelease Oct 6, 2020

ocefpaf deleted the 3.9rc1 branch October 6, 2020 17:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.9.0rc recipe #389

Python 3.9.0rc recipe #389

ocefpaf commented Sep 16, 2020

conda-forge-linter commented Sep 16, 2020

ocefpaf Sep 16, 2020

mingwandroid Sep 19, 2020 •

edited

Loading

mingwandroid Oct 2, 2020

willingc commented Sep 16, 2020

ocefpaf commented Sep 19, 2020

mingwandroid commented Sep 19, 2020

mingwandroid commented Sep 19, 2020

duncanmmacleod commented Sep 21, 2020

ocefpaf commented Sep 21, 2020

mingwandroid commented Sep 21, 2020

isuruf commented Sep 24, 2020

isuruf Sep 25, 2020

isuruf Sep 25, 2020

mingwandroid commented Sep 25, 2020

conda-forge-linter commented Oct 5, 2020

conda-forge-linter commented Oct 5, 2020

mingwandroid commented Oct 6, 2020

mingwandroid commented Oct 6, 2020

conda-forge-linter commented Oct 6, 2020

mingwandroid commented Oct 6, 2020

ocefpaf commented Oct 6, 2020

ocefpaf commented Oct 6, 2020

github-actions bot commented Oct 6, 2020

mingwandroid commented Oct 6, 2020

Python 3.9.0rc recipe #389

Python 3.9.0rc recipe #389

Conversation

ocefpaf commented Sep 16, 2020

conda-forge-linter commented Sep 16, 2020

ocefpaf Sep 16, 2020

Choose a reason for hiding this comment

mingwandroid Sep 19, 2020 • edited Loading

Choose a reason for hiding this comment

mingwandroid Oct 2, 2020

Choose a reason for hiding this comment

willingc commented Sep 16, 2020

ocefpaf commented Sep 19, 2020

mingwandroid commented Sep 19, 2020

mingwandroid commented Sep 19, 2020

duncanmmacleod commented Sep 21, 2020

ocefpaf commented Sep 21, 2020

mingwandroid commented Sep 21, 2020

isuruf commented Sep 24, 2020

isuruf Sep 25, 2020

Choose a reason for hiding this comment

isuruf Sep 25, 2020

Choose a reason for hiding this comment

mingwandroid commented Sep 25, 2020

conda-forge-linter commented Oct 5, 2020

conda-forge-linter commented Oct 5, 2020

mingwandroid commented Oct 6, 2020

mingwandroid commented Oct 6, 2020

conda-forge-linter commented Oct 6, 2020

mingwandroid commented Oct 6, 2020

ocefpaf commented Oct 6, 2020

ocefpaf commented Oct 6, 2020

github-actions bot commented Oct 6, 2020

mingwandroid commented Oct 6, 2020

mingwandroid Sep 19, 2020 •

edited

Loading