Add a workflow to update dependencies #490

mayeut · 2020-12-20T16:11:40Z

Add a workflow to update dependencies.

Manual dispatch.

bin/update_dependencies.py

mayeut · 2020-12-20T16:24:25Z

The workflow does not appear in "Actions"... I must be doing something wrong but I have no clue at the moment.

henryiii · 2020-12-20T16:42:58Z

It may need to be in master - or it may have had to run once.

Run the update with the right file. Use current repo in the created link. Run on push for now (not in master yet to get the workflow_dispatch actually working).

mayeut · 2020-12-20T16:53:35Z

It may need to be in master - or it may have had to run once.

From what I've read around, probably needs to be in master. Adding on push for now (to trigger a sample PR)

mayeut · 2020-12-20T16:54:41Z

And here is the sample PR: #491

joerick · 2020-12-20T16:56:24Z

Nice!

Default method is to use docker

joerick · 2020-12-20T17:25:27Z

It's pretty cool that it updates the existing PR if there's already one open! I think once we remove the 'push' trigger this looks good, to me

PR are only created for master

mayeut · 2020-12-20T17:27:42Z

It's pretty cool that it updates the existing PR if there's already one open! I think once we remove the 'push' trigger this looks good, to me

I removed the push trigger and enforce PR creation to be done only on master (I can revert this last one if that's not wanted).

joerick

Apologies, I just realised the implicit Python that's generating the versionless constraints.txt should probably be controlled too.

.github/workflows/update-dependencies.yml

bin/update_dependencies.py

joerick · 2020-12-20T19:12:23Z

bin/update_dependencies.py

@@ -12,21 +13,39 @@

 # CUSTOM_COMPILE_COMMAND is a pip-compile option that tells users how to
 # regenerate the constraints files
-os.environ['CUSTOM_COMPILE_COMMAND'] = "bin/update_constraints.py"
+os.environ['CUSTOM_COMPILE_COMMAND'] = "bin/update_dependencies.py"
 subprocess.check_call([


I just realised... this command is implicitly run in python3.8, because that's what my venv is in on my mac >.< (maybe a little sloppy, sorry!). Could it be made explicit? Perhaps 38 and 39 should be added to the list of versions and the default constraints.txt uses the final entry of this list? Then it should also be generated inside the docker container, so we're sure which version of Python is running it.

Shall we use a default constraints.txt at all ?

That's a great point. Maybe we don't need it at all, in fact!

Oh, well actually our code does assert that it exists, currently

It's also maybe useful as a starting point if somebody wanted to customise their constraints. But just copying the latest Python we support should do

Done. I had to modify a test so I updated all dependencies. Anyway, I think I'll have to rebase on #489 anyway.

Co-authored-by: Joe Rickerby <[email protected]>

This also updates constraints in order to get something consistent given that one test will require to be updated.

bin/update_dependencies.py

YannickJadoul · 2020-12-20T20:32:11Z

unit_test/dependency_constraints_test.py

@@ -10,7 +10,8 @@ def test_defaults():
    resources_dir = project_root / 'cibuildwheel' / 'resources'

    assert dependency_constraints.base_file_path.samefile(resources_dir / 'constraints.txt')
-    assert dependency_constraints.get_for_python_version('3.8').samefile(resources_dir / 'constraints.txt')
+    assert dependency_constraints.get_for_python_version('3.99').samefile(resources_dir / 'constraints.txt')


Is there a reason to still have a default constraints.txt file?

I thought the reason it's there is because 3.8 and 3.9 currently have the exact same version file?

see @joerick comments #490 (comment) on his review.

Yeah, I saw, but to start customizing your own constraints, you could also start from constraints-python39.txt? constraints.txt feels like a historical remainder, somehow. But ofc, it's not wrong; I was just thinking if it's not better to get a reminder (in the form of a failing CI), when we add 3.10, and future versions to cibuildwheel?

I think it's simpler just to have one per version. "But just copying the latest Python we support should do" I think is exactly what @YannickJadoul is referring to here.

Yeah, we're coming from a situation at the start where we only had one constraints.txt. Then 2.7 started acting up, and we had constraints-python2.7.txt, then 3.5, then 3.6, ... and now, there seems to be the realization that just having a separate file for all is better.

So why not just ditch the "default", then, if falling back to the default is a mistake?

Not that crucial, though. Just wondering if there's a good reason. (There's nothing else to nitpick about, so I need to find something? I very much like the idea and rest of the PR :-) )

@YannickJadoul, @henryiii, I agree and proposed to do so. Final decision left to @joerick.

Right. Sorry; should have added my comment to that thread, then :-)

I'm happy to go with that - that we don't bundle a constraints.txt. I think as @YannickJadoul says, the version-specific ones appeared to solve specific problems, but version-specific is probably a more robust solution. We'll have to remove this line but that's not a huge deal, I think.

Hmmm, looking at that code, I'm now seeing it's because we want to keep supporting a default constraints.txt for users.

I'm fine either way, then! As you see fit, @mayeut. Sorry about the confusion!

mayeut · 2020-12-21T18:53:37Z

I merged as-is.
We can still update the bit about constraints.txt later if needed.

Thanks for your reviews.

mayeut · 2020-12-21T19:07:00Z

I can, now that it's in master, dispatch the workflow. #495 created (can probably be closed though).
@joerick expressed at some point that he was not a big fan of dependabot because of the "sea of PR" it was creating. If need be and if this is not seen as "nuisance" PR, the workflow could also be run as a cron job.

henryiii · 2020-12-21T21:43:59Z

Dependabot (referring to .github/dependabot.yml) is configurable - you can have it update once a week if you like. There have been a few other things under the name dependabot, including a built-in security updater. But the configurable one is quite nice.

joerick · 2020-12-23T19:02:45Z

I'd be happy for this to run regularly, perhaps weekly? It's always just one PR, as opposed to the dependabot 'sea', and we can dial it back if it becomes irritating.

…

On 21 Dec 2020, at 19:07, Matthieu Darbois ***@***.***> wrote: I can, now that it's in master, dispatch the workflow. #495 <#495> created (can probably be closed though). @joerick <https://github.com/joerick> expressed at some point <#256 (comment)> that he was not a big fan of dependabot because of the "sea of PR" it was creating. If need be and if this is not seen as "nuisance" PR, the workflow could also be run as a cron job. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#490 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJPZE4QVQX4KEHD26257DTSV6MGNANCNFSM4VDGAGAA>.

YannickJadoul · 2020-12-23T19:12:36Z

As noted by @henryiii (somewhere else; I forgot why), it might also be nice to update the Python versions?
The main question is: is there a clean way of getting a list of Python versions?

mayeut · 2020-12-23T19:22:02Z

@YannickJadoul, I don't know about "clean" but I have been experimenting with this because it's also something I'd like to have in manylinux. I expended the experiment for cibuildwheel and I will post a draft PR showing it can work. I'm not sure I will have time to improve on it though but it might give some ideas/pointers to others.

YannickJadoul · 2020-12-23T19:36:28Z

@mayeut That would be amazing :-) Anything that can help us get started could help, anyway.

joerick · 2020-12-23T19:37:16Z

I agree, it would be nice to automate the Python upgrades as well. Sounds good @mayeut , I look forward to seeing it! The only way I can think of to do it would be to parse https://www.python.org/ftp/python/ ...

YannickJadoul · 2020-12-23T19:39:07Z

Yes, by "clean", I was referring to something like PyPy's https://downloads.python.org/pypy/versions.json, and not having to parse things, or check if trying to download e.g. https://www.python.org/ftp/python/3.9.2/python-3.9.2-macos11.0.pkg doesn't fail.

mayeut · 2020-12-23T19:49:42Z

Draft PR #496 created.
@YannickJadoul, the ideas implemented are "clean" per your criteria, the implementation is not.

Add a workflow to update dependencies

336db95

henryiii reviewed Dec 20, 2020

View reviewed changes

bin/update_dependencies.py Outdated Show resolved Hide resolved

Fix a few things in update workflow

3b465d8

Run the update with the right file. Use current repo in the created link. Run on push for now (not in master yet to get the workflow_dispatch actually working).

Use '--no-docker' to run dependency update using local pythons

17798c5

Default method is to use docker

Only run the dependency update workflow manually

5ffc7e4

PR are only created for master

mayeut marked this pull request as ready for review December 20, 2020 17:26

joerick reviewed Dec 20, 2020

View reviewed changes

mayeut and others added 2 commits December 20, 2020 20:31

Update .github/workflows/update-dependencies.yml

55048c7

Co-authored-by: Joe Rickerby <[email protected]>

Create constraints for all python versions

13b7ed3

This also updates constraints in order to get something consistent given that one test will require to be updated.

YannickJadoul reviewed Dec 20, 2020

View reviewed changes

bin/update_dependencies.py Outdated Show resolved Hide resolved

YannickJadoul reviewed Dec 20, 2020

View reviewed changes

Use manylinux2010_x86_64:latest to update dependencies

d0b61cd

YannickJadoul approved these changes Dec 21, 2020

View reviewed changes

henryiii approved these changes Dec 21, 2020

View reviewed changes

mayeut merged commit 40ddede into master Dec 21, 2020

mayeut deleted the update-dependencies-workflow branch December 21, 2020 18:58

mayeut mentioned this pull request Dec 23, 2020

feat: update python versions as part of update_dependencies #496

Merged

mayeut mentioned this pull request Dec 23, 2020

Run update workflow weekly #497

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a workflow to update dependencies #490

Add a workflow to update dependencies #490

mayeut commented Dec 20, 2020

mayeut commented Dec 20, 2020

henryiii commented Dec 20, 2020

mayeut commented Dec 20, 2020

mayeut commented Dec 20, 2020

joerick commented Dec 20, 2020

joerick commented Dec 20, 2020

mayeut commented Dec 20, 2020

joerick left a comment

joerick Dec 20, 2020

mayeut Dec 20, 2020

joerick Dec 20, 2020

joerick Dec 20, 2020

joerick Dec 20, 2020

mayeut Dec 20, 2020

YannickJadoul Dec 20, 2020

mayeut Dec 20, 2020

YannickJadoul Dec 20, 2020

henryiii Dec 20, 2020

YannickJadoul Dec 20, 2020

YannickJadoul Dec 20, 2020

mayeut Dec 20, 2020

YannickJadoul Dec 20, 2020

joerick Dec 21, 2020

YannickJadoul Dec 21, 2020

mayeut commented Dec 21, 2020

mayeut commented Dec 21, 2020

henryiii commented Dec 21, 2020

joerick commented Dec 23, 2020 via email

YannickJadoul commented Dec 23, 2020

mayeut commented Dec 23, 2020

YannickJadoul commented Dec 23, 2020

joerick commented Dec 23, 2020

YannickJadoul commented Dec 23, 2020 •

edited

Loading

mayeut commented Dec 23, 2020 •

edited

Loading

Add a workflow to update dependencies #490

Add a workflow to update dependencies #490

Conversation

mayeut commented Dec 20, 2020

mayeut commented Dec 20, 2020

henryiii commented Dec 20, 2020

mayeut commented Dec 20, 2020

mayeut commented Dec 20, 2020

joerick commented Dec 20, 2020

joerick commented Dec 20, 2020

mayeut commented Dec 20, 2020

joerick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayeut commented Dec 21, 2020

mayeut commented Dec 21, 2020

henryiii commented Dec 21, 2020

joerick commented Dec 23, 2020 via email

YannickJadoul commented Dec 23, 2020

mayeut commented Dec 23, 2020

YannickJadoul commented Dec 23, 2020

joerick commented Dec 23, 2020

YannickJadoul commented Dec 23, 2020 • edited Loading

mayeut commented Dec 23, 2020 • edited Loading

YannickJadoul commented Dec 23, 2020 •

edited

Loading

mayeut commented Dec 23, 2020 •

edited

Loading