Skip to content

Commit

Permalink
Arrow connector workspace (#31)
Browse files Browse the repository at this point in the history 
* Arrow Flight connector template

Co-authored-by: sai bhaskar reddy <[email protected]>
Co-authored-by: SthuthiGhosh9400 <[email protected]>
Co-authored-by: lithinwxd <[email protected]>
Co-authored-by: Steve Burnett <[email protected]>
Co-authored-by: elbinpallimalilibm <[email protected]>

* Move Arrow type to Presto type mapping to a separate function

* Fix checkstyle errors

* Refactor null check

* Remove explicit versios

* Remove explicit versions

* Remove explicit versions in pom (#25)

* Remove explicit versios

* Remove explicit versions

* Remove explicit declarations in pom

* Resolve upper bound error

* Do not exclude netty dependencies

* Remove explicit versions from pom

* Add integration smoke test

* Use flight descriptor instead of ArrowFlightRequest

* Remove config getter from flight client handler

* Arrow connector reviewcomment fixes (#28)

* Arrow CI job

* added verison in property file

* testing after removing hyphen

* remove path to make sure CI is run during every build

* changed yaml to run CI on evry pull requests and every update except doc update

* review comment fixes

* Use flight descriptor instead of ArrowFlightRequest

* Arrow page utils changes

* Use flight descriptor instead of ArrowFlightRequest

* Arrow page utils changes - fixed checkstyle issues

* Review comment fixes - Root allocator and typos

* Remove config getter from flight client handler

* Arrow CI job

* added verison in property file

* testing after removing hyphen

* remove path to make sure CI is run during every build

* changed yaml to run CI on evry pull requests and every update except doc update

* review comment fixes

* Arrow page utils changes

* Arrow page utils changes - fixed checkstyle issues

* Review comment fixes - Root allocator and typos

* Review comment fixes

* Review comment fixes

* Review comment fixes - Changed config

* Added support for small int tiny int date and timestamp

* Review comment fixes - Dictionary encoding and other tests

* Review comment fixes - Added Tests as per comments

* Removed license header unwanted place

* Removed duplicate CI job

* Review comment fixes

* Added more testcases

* Fixed review comments and added support for other datatypes

* Minor fixes on method argument

* Fixed review comments - cosmetic changes and unneccessary class removal

* Arrow - DictionaryEncoding usecase

* Removed description which looks like generated

* Added support for dictionary encoding

* Fixed review comments

* Fixed review comments -added index type

---------

Co-authored-by: Elbin Pallimalil <[email protected]>
Co-authored-by: Elbin Pallimalil <[email protected]>

---------

Co-authored-by: sai bhaskar reddy <[email protected]>
Co-authored-by: SthuthiGhosh9400 <[email protected]>
Co-authored-by: lithinwxd <[email protected]>
Co-authored-by: Steve Burnett <[email protected]>
  • Loading branch information
@elbinpallimalilibm @sabbasani
@SthuthiGhosh9400 @lithinwxd @steveburnett
5 people authored Dec 9, 2024
1 parent f38c212 commit 13ac221
Show file tree
Hide file tree
Showing 31 changed files with 2,621 additions and 843 deletions.
82 changes: 82 additions & 0 deletions .github/workflows/arrow-flight-tests.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
name: arrow flight tests

on:
pull_request:

env:
CONTINUOUS_INTEGRATION: true
MAVEN_OPTS: "-Xmx1024M -XX:+ExitOnOutOfMemoryError"
MAVEN_INSTALL_OPTS: "-Xmx2G -XX:+ExitOnOutOfMemoryError"
MAVEN_FAST_INSTALL: "-B -V --quiet -T 1C -DskipTests -Dair.check.skip-all --no-transfer-progress -Dmaven.javadoc.skip=true"
MAVEN_TEST: "-B -Dair.check.skip-all -Dmaven.javadoc.skip=true -DLogTestDurationListener.enabled=true --no-transfer-progress --fail-at-end"
RETRY: .github/bin/retry

jobs:
changes:
runs-on: ubuntu-latest
permissions:
pull-requests: read
outputs:
codechange: ${{ steps.filter.outputs.codechange }}
steps:
- uses: dorny/paths-filter@v2
id: filter
with:
filters: |
codechange:
- '!presto-docs/**'
test:
runs-on: ubuntu-latest
needs: changes
strategy:
fail-fast: false
matrix:
modules:
- ":presto-base-arrow-flight" # Only run tests for the `presto-base-arrow-flight` module

timeout-minutes: 80
concurrency:
group: ${{ github.workflow }}-test-${{ matrix.modules }}-${{ github.event.pull_request.number }}
cancel-in-progress: true

steps:
# Checkout the code only if there are changes in the relevant files
- uses: actions/checkout@v4
if: needs.changes.outputs.codechange == 'true'
with:
show-progress: false

# Set up Java for the build environment
- uses: actions/setup-java@v2
if: needs.changes.outputs.codechange == 'true'
with:
distribution: 'temurin'
java-version: 8

# Cache Maven dependencies to speed up the build
- name: Cache local Maven repository
if: needs.changes.outputs.codechange == 'true'
id: cache-maven
uses: actions/cache@v2
with:
path: ~/.m2/repository
key: ${{ runner.os }}-maven-2-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-2-
# Resolve Maven dependencies (if cache is not found)
- name: Populate Maven cache
if: steps.cache-maven.outputs.cache-hit != 'true' && needs.changes.outputs.codechange == 'true'
run: ./mvnw de.qaware.maven:go-offline-maven-plugin:resolve-dependencies --no-transfer-progress && .github/bin/download_nodejs

# Install dependencies for the target module
- name: Maven Install
if: needs.changes.outputs.codechange == 'true'
run: |
export MAVEN_OPTS="${MAVEN_INSTALL_OPTS}"
./mvnw install ${MAVEN_FAST_INSTALL} -am -pl ${{ matrix.modules }}
# Run Maven tests for the target module
- name: Maven Tests
if: needs.changes.outputs.codechange == 'true'
run: ./mvnw test ${MAVEN_TEST} -pl ${{ matrix.modules }}
142 changes: 47 additions & 95 deletions presto-base-arrow-flight/pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,14 @@
<dep.okhttp.version>4.10.0</dep.okhttp.version>
<arrow.version>17.0.0</arrow.version>
<netty.version>4.1.110.Final</netty.version>
<kotlin.version>1.6.20</kotlin.version>
<error_prone_annotations>2.23.0</error_prone_annotations>
</properties>

<dependencies>
<dependency>
<groupId>com.facebook.airlift</groupId>
<artifactId>bootstrap</artifactId>
<exclusions>
<exclusion>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
Expand All @@ -39,14 +35,7 @@
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<exclusions>
<exclusion>
<groupId>org.checkerframework</groupId>
<artifactId>checker-qual</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.errorprone</groupId>
<artifactId>error_prone_annotations</artifactId>
</exclusion>
<!-- Excluding to resolve upper bound errors -->
<exclusion>
<groupId>com.google.j2objc</groupId>
<artifactId>j2objc-annotations</artifactId>
Expand Down Expand Up @@ -99,6 +88,12 @@
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-memory-core</artifactId>
<version>${arrow.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
Expand All @@ -115,37 +110,29 @@
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-jdbc</artifactId>
<version>${arrow.version}</version>
</dependency>

<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-codec-http2</artifactId>
<version>${netty.version}</version>
</dependency>

<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-handler-proxy</artifactId>
<version>${netty.version}</version>
<exclusions>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-codec-http</artifactId>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>2.0.65.Final</version>
</dependency>

<dependency>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-vector</artifactId>
<version>${arrow.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<!-- Excluding to resolve upper bound errors -->
<exclusion>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</exclusion>
<!-- Excluding to resolve upper bound errors -->
<exclusion>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
Expand Down Expand Up @@ -188,13 +175,6 @@
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.checkerframework</groupId>
<artifactId>checker-qual</artifactId>
<version>3.4.1</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-testng-services</artifactId>
Expand Down Expand Up @@ -225,6 +205,11 @@
<artifactId>flight-core</artifactId>
<version>${arrow.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
<!-- Excluding to resolve upper bound errors -->
<exclusion>
<groupId>com.google.j2objc</groupId>
<artifactId>j2objc-annotations</artifactId>
Expand All @@ -249,41 +234,10 @@
<artifactId>h2</artifactId>
<scope>test</scope>
</dependency>

</dependencies>

<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.codehaus.mojo</groupId>
<artifactId>animal-sniffer-annotations</artifactId>
<version>1.23</version>
</dependency>

<dependency>
<groupId>com.google.j2objc</groupId>
<artifactId>j2objc-annotations</artifactId>
<version>1.3</version> <!-- Or the version you want to enforce -->
</dependency>

<dependency>
<groupId>com.google.errorprone</groupId>
<artifactId>error_prone_annotations</artifactId>
<version>2.14.0</version> <!-- Use the highest version found in conflict -->
</dependency>

<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.25.5</version> <!-- Use the version that resolves conflicts -->
</dependency>

<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.17.0</version>
</dependency>

<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-unix-common</artifactId>
Expand Down Expand Up @@ -321,22 +275,36 @@
</dependency>

<dependency>
<groupId>org.objenesis</groupId>
<artifactId>objenesis</artifactId>
<version>3.3</version>
<groupId>io.netty</groupId>
<artifactId>netty-handler-proxy</artifactId>
<version>${netty.version}</version>
</dependency>

<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-codec-http</artifactId>
<version>${netty.version}</version>
</dependency>

<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-common</artifactId>
<version>1.6.20</version>
<version>${kotlin.version}</version>
</dependency>

<dependency>
<groupId>com.google.errorprone</groupId>
<artifactId>error_prone_annotations</artifactId>
<version>${error_prone_annotations}</version>
</dependency>

<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>2.0.13</version>
<groupId>org.apache.arrow</groupId>
<artifactId>arrow-algorithm</artifactId>
<version>${arrow.version}</version>
<scope>compile</scope>
</dependency>

</dependencies>
</dependencyManagement>

Expand All @@ -352,26 +320,10 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<configuration>
<rules>
<requireUpperBoundDeps>
<excludes combine.children="append">
<exclude>com.google.errorprone:error_prone_annotations</exclude>
</excludes>
</requireUpperBoundDeps>
</rules>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<configuration>
<ignoredUnusedDeclaredDependencies>
<ignoredUnusedDeclaredDependency>io.netty:netty-codec-http2</ignoredUnusedDeclaredDependency>
<ignoredUnusedDeclaredDependency>io.netty:netty-handler-proxy</ignoredUnusedDeclaredDependency>
<ignoredUnusedDeclaredDependency>io.netty:netty-tcnative-boringssl-static</ignoredUnusedDeclaredDependency>
</ignoredUnusedDeclaredDependencies>
</configuration>
</plugin>
<plugin>
<groupId>org.basepom.maven</groupId>
Expand Down
Loading

0 comments on commit 13ac221

Please sign in to comment.