[GLUTEN-4903][CELEBORN] Support multiple versions of Celeborn #4913
Conversation
|
Run Gluten Clickhouse CI |
1 similar comment
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Is it possible to use one module to be compatible with multi-version celeborn ? It is hard to maintain if we always create new module for new coming celeborn version. Besides, it's hard to work if someone use 0.3 branch but backport some 0.4 patch.. cc @pan3793 |
@ulysses-you I do agree that we need to consider the case of |
kerwin-zk
changed the title
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
1 similar comment
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
3 similar comments
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
kerwin-zk
force-pushed
the
issue-4903
branch
2 times, most recently
from
0678ce5 to
56cda5b
Compare
|
Run Gluten Clickhouse CI |
1 similar comment
|
Run Gluten Clickhouse CI |
kerwin-zk
changed the title
| run: | | ||
| $PATH_TO_GLUTEN_TE/$OS_IMAGE_NAME/gha/gha-checkout/exec.sh \ | ||
| 'wget https://archive.apache.org/dist/incubator/celeborn/celeborn-0.3.0-incubating/apache-celeborn-0.3.0-incubating-bin.tgz && \ | ||
| tar xzf apache-celeborn-0.3.0-incubating-bin.tgz -C /opt/ && mv /opt/apache-celeborn-0.3.0-incubating-bin /opt/celeborn && cd /opt/celeborn && \ | ||
| 'wget https://archive.apache.org/dist/incubator/celeborn/celeborn-0.4.0-incubating/apache-celeborn-0.4.0-incubating-bin.tgz && \ |
There was a problem hiding this comment.
CI only covers 0.3.2 and 0.4.0, what about 0.3.0 and 0.3.1? we actually have breaking changes on the APIs that Gluten used in the 0.3.x serial
There was a problem hiding this comment.
@pan3793 CI of 0.3.0 and 0.3.1 has been successfully run. Mainly considering the running time of CI, only one version of 0.3.x is currently retained. If everyone feels that CI duration is not an issue after discussion, I can add it.
| <exclusions> | ||
| <exclusion> | ||
| <groupId>org.apache.celeborn</groupId> | ||
| <artifactId>celeborn-client-spark-${spark.major.version}_${scala.binary.version}</artifactId> | ||
| </exclusion> | ||
| <exclusion> | ||
| <groupId>org.apache.celeborn</groupId> | ||
| <artifactId>celeborn-spark-${spark.major.version}-columnar-shuffle_${scala.binary.version}</artifactId> | ||
| </exclusion> | ||
| </exclusions> |
There was a problem hiding this comment.
the shaded artifacts should not expose the shipped transitive deps, seems there are issues in 0.4.0-incubating cc @cfmcgrady
There was a problem hiding this comment.
@pan3793 I am handling this issue.
|
Run Gluten Clickhouse CI |
| @@ -47,7 +47,7 @@ | |||
| <sparkbundle.version>3.2</sparkbundle.version> | |||
| <spark.version>3.4.2</spark.version> | |||
| <sparkshim.artifactId>spark-sql-columnar-shims-spark32</sparkshim.artifactId> | |||
| <celeborn.version>0.3.0-incubating</celeborn.version> | |||
| <celeborn.version>0.3.2-incubating</celeborn.version> | |||
There was a problem hiding this comment.
shall we add profile for 0.4 ?
There was a problem hiding this comment.
In theory, Celeborn only needs one version, as differences between versions have been eliminated through reflection.
...rn/common/src/main/java/org/apache/spark/shuffle/gluten/celeborn/CelebornShuffleManager.java
Outdated
Show resolved
Hide resolved
...rn/common/src/main/java/org/apache/spark/shuffle/gluten/celeborn/CelebornShuffleManager.java
Outdated
Show resolved
Hide resolved
...rn/common/src/main/java/org/apache/spark/shuffle/gluten/celeborn/CelebornShuffleManager.java
Outdated
Show resolved
Hide resolved
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
|
What changes were proposed in this pull request?
Support multiple versions of Celeborn
#4903
How was this patch tested?
CI