[NSE-304] Upgrade to Arrow 4.0.0 #307
Conversation
|
Native SQL engine will be based on https://github.com/oap-project/arrow/tree/arrow-4.0.0-oap after applying this. |
zhztheplayer
force-pushed
the
NSE-304
branch
2 times, most recently
from
657a8b4 to
1b436f3
Compare
|
@github-actions ram-usage-test |
zhztheplayer
force-pushed
the
NSE-304
branch
2 times, most recently
from
1c3d262 to
7d10fea
Compare
| <artifactId>log4j</artifactId> | ||
| </exclusion> | ||
| </exclusions> | ||
| <artifactId>hadoop-client</artifactId> |
There was a problem hiding this comment.
not sure if this will break v2 @rui-mo
There was a problem hiding this comment.
If executions are required they were added to the parent pom https://github.com/oap-project/native-sql-engine/pull/307/files/7a39548a3dedc3c8f89127f4cac712a4c20864c6#diff-9c5fb3d1b7e3b0f54bc5c4182965c4fe1f9023d449017cece3005d3f90e8e4d8R145-R170
There was a problem hiding this comment.
And is there other reason v2 is breaked?
There was a problem hiding this comment.
the issue are mostly due to spark pre-build binary with hadoop 3.2 - which packages some specific version of depdencies other than those in hadoop-common-3.2.0. will do a quick test on my env to verify
https://github.com/apache/spark/blob/v3.1.1/pom.xml#L172
https://github.com/apache/spark/blob/v3.1.1/pom.xml#L121
There was a problem hiding this comment.
@zhouyuan it is ok on my env
|
|
|
|
|
|
|
thanks @zhztheplayer ! |
* [NSE-304] Upgrade to Arrow 4.0.0 * use .400 * .400 * fix * fix
* [NSE-304] Upgrade to Arrow 4.0.0 (#307) * [NSE-304] Upgrade to Arrow 4.0.0 * use .400 * .400 * fix * fix * [NSE-318]check schema before reuse exchange (#328) * check schema before reuse exchange Signed-off-by: Yuan Zhou <[email protected]> * remove debug log Signed-off-by: Yuan Zhou <[email protected]> * bump version to 1.1.1 Signed-off-by: Yuan Zhou <[email protected]> * update arrow branch to use arrow-4.0.0-oap-1.1.1 Signed-off-by: Yuan Zhou <[email protected]> * update GHA arrow branch Signed-off-by: Yuan Zhou <[email protected]> * [NSE-329] fix out partitioning in BHJ and SHJ (#335) * fix out partitioning for SHJ and BHJ * refine * [NSE-325] fix incremental compile issue with 4.5.x scala-maven-plugin (#332) the new release of scala-maven-plugin(4.5.x) seems buggy on incremental compile, using 4.4.0 instead Signed-off-by: Yuan Zhou <[email protected]> Co-authored-by: Hongze Zhang <[email protected]> Co-authored-by: Rui Mo <[email protected]>
No description provided.