Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Ray] Enable CI of mars/learn for Ray DAG #3261

Merged
merged 27 commits into from
Oct 12, 2022
Merged

[Ray] Enable CI of mars/learn for Ray DAG #3261

merged 27 commits into from
Oct 12, 2022

Conversation

fyrestone
Copy link
Contributor

@fyrestone fyrestone commented Sep 19, 2022 •
edited
Loading

What do these changes do?

Enables CI of mars/learn for Ray DAG.

  • [shuffle] Fix Ray executor failed to run learn shuffle.
  • [shuffle] Fix Ray executor shuffle colouring bug.
  • [shuffle] Fix Ray executor bincount shuffle.
  • Fix Ray executor raises KeyError when exiting stage.
  • Fix Ray executor raises ValueError when running learn LabelEncoder.
  • Fix Ray executor raises ValueError: WRITEBACKIFCOPY base is read-only.
  • Fix Ray executor raises ValueError: Tileable object must be executed first before fetch.
  • Fix ParallelPostFit raises AttributeError: 'GradientBoostingClassifier' object has no attribute '_loss'.
  • Fix Ray executor run test_classification.test__check_targets may raise KeyError.
  • Fix Ray executor init ray twice?

Related issue number

Fixes:
#3260
#3262
#3256
#3267
#3268
#3269
#3266
#3270
#3272
#3274

Related: #3239

Check code requirements

  • tests added / passed (if needed)
  • Ensure all linting tests pass, see here for how to run them

@fyrestone fyrestone self-assigned this Sep 19, 2022
@fyrestone fyrestone changed the title [Ray] Fix Ray executor learn shuffle [Ray] Fix Ray executor run learn shuffle failed Sep 19, 2022
@fyrestone fyrestone marked this pull request as ready for review September 19, 2022 10:49
@fyrestone fyrestone requested a review from a team as a code owner September 19, 2022 10:49
@fyrestone fyrestone changed the title [Ray] Fix Ray executor run learn shuffle failed [Ray] Fix Ray executor run mars learn bugs Sep 21, 2022
@fyrestone fyrestone changed the title [Ray] Fix Ray executor run mars learn bugs [Ray] Enable CI of mars/learn for Ray DAG Sep 22, 2022
chaokunyang
chaokunyang previously approved these changes Oct 9, 2022
View reviewed changes
Copy link
Contributor

@chaokunyang chaokunyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

zhongchun
zhongchun approved these changes Oct 12, 2022
View reviewed changes
Copy link
Contributor

@zhongchun zhongchun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

chaokunyang
chaokunyang approved these changes Oct 12, 2022
View reviewed changes
Copy link
Contributor

@chaokunyang chaokunyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chaokunyang chaokunyang merged commit 3a0a99c into mars-project:master Oct 12, 2022
qianduoduo0904 pushed a commit to qianduoduo0904/mars that referenced this pull request Oct 13, 2022
@fyrestone @qianduoduo0904
[Ray] Enable CI of mars/learn for Ray DAG (mars-project#3261)
073c516 
* Fix Ray DAG learn shuffle

* Fix special cases that some meta has been updated in previous stages

* Enable tests and remove init metrics

* Fix Ray executor can't run learn LabelEncoder

* Fix TensorPower on sparse matrix raises WRITEBACKIFCOPY base is read-only

* Fix check_binarized_results raises WRITEBACKIFCOPY base is read-only

* Fix shuffle colouring bug

* RemoteFunction support resolve_tileable_input

* Enable CI of mars/learn

* Remove @pytest.mark.ray_dag

* Pin pandas<1.5.0

* Fix

* Fix bincount shuffle

* Log not init error once

* Fix Ray executor result tileable ref

* Remove ensure_coverage() in ray executor

* Fix

* Fix ray.init twice?

* Try to fix Ray DAG coverage

* Improve coverage

* Coverage

* Fix

* Revert "Coverage"

This reverts commit 4174168.

* Revert "Try to fix Ray DAG coverage"

This reverts commit f7e8276.

* Coverage

Co-authored-by: 刘宝 <[email protected]>
(cherry picked from commit 3a0a99c)
qianduoduo0904 pushed a commit to qianduoduo0904/mars that referenced this pull request Oct 13, 2022
@fyrestone @qianduoduo0904
[Ray] Enable CI of mars/learn for Ray DAG (mars-project#3261)
c0b9506 
* Fix Ray DAG learn shuffle

* Fix special cases that some meta has been updated in previous stages

* Enable tests and remove init metrics

* Fix Ray executor can't run learn LabelEncoder

* Fix TensorPower on sparse matrix raises WRITEBACKIFCOPY base is read-only

* Fix check_binarized_results raises WRITEBACKIFCOPY base is read-only

* Fix shuffle colouring bug

* RemoteFunction support resolve_tileable_input

* Enable CI of mars/learn

* Remove @pytest.mark.ray_dag

* Pin pandas<1.5.0

* Fix

* Fix bincount shuffle

* Log not init error once

* Fix Ray executor result tileable ref

* Remove ensure_coverage() in ray executor

* Fix

* Fix ray.init twice?

* Try to fix Ray DAG coverage

* Improve coverage

* Coverage

* Fix

* Revert "Coverage"

This reverts commit 4174168.

* Revert "Try to fix Ray DAG coverage"

This reverts commit f7e8276.

* Coverage

Co-authored-by: 刘宝 <[email protected]>
(cherry picked from commit 3a0a99c)
aresnow1 pushed a commit to xorbitsai/mars that referenced this pull request Oct 13, 2022
@fyrestone @aresnow1
[Ray] Enable CI of mars/learn for Ray DAG (mars-project#3261)
22c737c 
* Fix Ray DAG learn shuffle

* Fix special cases that some meta has been updated in previous stages

* Enable tests and remove init metrics

* Fix Ray executor can't run learn LabelEncoder

* Fix TensorPower on sparse matrix raises WRITEBACKIFCOPY base is read-only

* Fix check_binarized_results raises WRITEBACKIFCOPY base is read-only

* Fix shuffle colouring bug

* RemoteFunction support resolve_tileable_input

* Enable CI of mars/learn

* Remove @pytest.mark.ray_dag

* Pin pandas<1.5.0

* Fix

* Fix bincount shuffle

* Log not init error once

* Fix Ray executor result tileable ref

* Remove ensure_coverage() in ray executor

* Fix

* Fix ray.init twice?

* Try to fix Ray DAG coverage

* Improve coverage

* Coverage

* Fix

* Revert "Coverage"

This reverts commit 4174168.

* Revert "Try to fix Ray DAG coverage"

This reverts commit f7e8276.

* Coverage

Co-authored-by: 刘宝 <[email protected]>
(cherry picked from commit 3a0a99c)
qianduoduo0904 pushed a commit to qianduoduo0904/mars that referenced this pull request Oct 24, 2022
@fyrestone @qianduoduo0904
[Ray] Enable CI of mars/learn for Ray DAG (mars-project#3261)
fccf4ac 
* Fix Ray DAG learn shuffle

* Fix special cases that some meta has been updated in previous stages

* Enable tests and remove init metrics

* Fix Ray executor can't run learn LabelEncoder

* Fix TensorPower on sparse matrix raises WRITEBACKIFCOPY base is read-only

* Fix check_binarized_results raises WRITEBACKIFCOPY base is read-only

* Fix shuffle colouring bug

* RemoteFunction support resolve_tileable_input

* Enable CI of mars/learn

* Remove @pytest.mark.ray_dag

* Pin pandas<1.5.0

* Fix

* Fix bincount shuffle

* Log not init error once

* Fix Ray executor result tileable ref

* Remove ensure_coverage() in ray executor

* Fix

* Fix ray.init twice?

* Try to fix Ray DAG coverage

* Improve coverage

* Coverage

* Fix

* Revert "Coverage"

This reverts commit 4174168.

* Revert "Try to fix Ray DAG coverage"

This reverts commit f7e8276.

* Coverage

Co-authored-by: 刘宝 <[email protected]>
(cherry picked from commit 3a0a99c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhongchun zhongchun zhongchun approved these changes

@chaokunyang chaokunyang chaokunyang approved these changes

Assignees

@fyrestone fyrestone

Labels
mod: learn mod: ray integration shuffle
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@fyrestone @zhongchun @chaokunyang