Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci/docker/air] Update ML/DL dependencies to latest releases / Remove Py3.6 Docker images #28808

Merged
merged 39 commits into from
Oct 27, 2022

Conversation

krfricke
Copy link
Contributor

@krfricke krfricke commented Sep 27, 2022

Why are these changes needed?

This PR upgrades some of our dependencies to the latest releases.

At the moment, we are testing with a lower bound of dependencies. This is good as we ensure compatibility with these versions, but at the same time we don't test compatibility with more recent releases. We prioritize backwards compatibility over compatibility with more recent versions.

This PR:

  • Introduces a set of legacy dependencies (the current versions in the CI).
  • This is a lower bound of dependencies that we guarantee compatibility with
  • The regular dependencies are upgraded to more recent releases
  • We remove Docker builds for Python 3.6. as this is incompatible with more recent versions of tensorflow/torch
  • Wheel building for 3.6 is not affected.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Kai Fricke added 4 commits September 27, 2022 11:11
@krfricke krfricke changed the title [wip][ci/air] Update ML/DL dependencies to latest releases [wip][ci/air] Update ML/DL dependencies to latest releases / Remove Py3.6 Docker images Sep 27, 2022
Kai Fricke added 19 commits September 27, 2022 11:57
Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
tfp
Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Kai Fricke <[email protected]>
…endencies

# Conflicts:
#	python/requirements/ml/requirements_rllib.txt
Signed-off-by: Kai Fricke <[email protected]>
@krfricke krfricke marked this pull request as ready for review October 11, 2022 16:24
@krfricke
Copy link
Contributor Author

Tests are passing except for some in RLlib - I will work with the rllib team to resolve this.

But the rest is ready for review

cc @pcmoritz @richardliaw for general testing infrastructure
cc @gjoliver @sven1977 for rllib compatibility
cc @amogkam for air compatibility and dependencies

The base CI GPU image has been updated to support CUDA 11.6 already.

@ddelange
Copy link
Contributor

I guess this PR could incorporate, supercede and close #26072?

# Conflicts:
#	docker/ray-ml/Dockerfile
#	python/requirements/ml/requirements_tune.txt
@krfricke krfricke changed the title [wip][ci/air] Update ML/DL dependencies to latest releases / Remove Py3.6 Docker images [ci/docker/air] Update ML/DL dependencies to latest releases / Remove Py3.6 Docker images Oct 20, 2022
@krfricke
Copy link
Contributor Author

I guess this PR could incorporate, supercede and close #26072?

@ddelange how about we do this in two steps - merge this first and land your changes once pyg-team/pyg-lib#66 is addressed?

@ddelange
Copy link
Contributor

totally fine by me :)

@krfricke krfricke requested a review from amogkam October 21, 2022 15:15
Copy link
Contributor

@amogkam amogkam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @krfricke, lgtm!

@ddelange ddelange mentioned this pull request Oct 22, 2022
Signed-off-by: Artur Niederfahrenhorst <[email protected]>
Kai Fricke added 2 commits October 26, 2022 14:33
Signed-off-by: Kai Fricke <[email protected]>
# Conflicts:
#	rllib/tests/run_regression_tests.py
@@ -27,7 +27,7 @@ def test_a3c_compilation(self):
num_iterations = 2

# Test against all frameworks.
for _ in framework_iterator(config, with_eager_tracing=True):
for _ in framework_iterator(config, with_eager_tracing=False):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gjoliver @kouroshHakha Not sure why this test did not fail initially, but A3C seems error with eager tracing as well, right @krfricke ?

@richardliaw richardliaw removed their assignment Oct 27, 2022
@krfricke krfricke merged commit 6b9a56d into ray-project:master Oct 27, 2022
@krfricke krfricke deleted the ci/air-update-dependencies branch October 27, 2022 22:39
WeichenXu123 pushed a commit to WeichenXu123/ray that referenced this pull request Dec 19, 2022
… Py3.6 Docker images (ray-project#28808)

This PR upgrades some of our dependencies to the latest releases.

At the moment, we are testing with a lower bound of dependencies. This is good as we ensure compatibility with these versions, but at the same time we don't test compatibility with more recent releases. We prioritize backwards compatibility over compatibility with more recent versions.

This PR:
- Introduces a set of _legacy dependencies_ (the current versions in the CI).
- This is a lower bound of dependencies that we guarantee compatibility with
- The regular dependencies are upgraded to more recent releases
- We remove Docker builds for Python 3.6. as this is incompatible with more recent versions of tensorflow/torch
- Wheel building for 3.6 is not affected.

Signed-off-by: Kai Fricke <[email protected]>
Signed-off-by: Artur Niederfahrenhorst <[email protected]>
Co-authored-by: Artur Niederfahrenhorst <[email protected]>
Signed-off-by: Weichen Xu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants