Skip to content

Commit

Permalink
Buffer and coverage 21.2.x (#24)
Browse files Browse the repository at this point in the history
* Additional Functionality (#13) (#14)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Jts upgrade 21.1 (#18)

* Additional Functionality (#13)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Work on angle functionality and proj4 transform

* Added missing files

* Added ST_Angle implementations

* Added Azimuth

* Added ST_Union Aggregate variant

* Some fixes following manual tests

* Bad version fix

* 21.1.x add tests (#20)

* Dremio 21.2.x upgrade

* Updated WIKI link

* Additional Functionality (#13)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Additional Functionality (#13) (#15)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Work on angle functionality and proj4 transform

* Added missing files

* Added ST_Angle implementations

* Added Azimuth

* Added ST_Union Aggregate variant

* Some fixes following manual tests

* Readme Update

* version bump, updated readme

* ST_X tests foundations laid down

* Checkstyle IDEA

* Made improvement with test coverage

* A lot more work on tests
Better code formatting

* Some more relation tests cleanup

* Added detekt style checks and fixed CI

* Missing detekt plugin configuration

* Added output tests

* Add Geometry Input Tests

* Add Geometry input testing

* Added WKB Input tests

* Measurement functions testing

* Added overlay function tests

* Some more overlay function tests

* Added codecov

* Added GH hosted runners

* Coverall migration

* Coverall migration

* Coverall migration

* Coverall migration

* Coverall migration

* Added some more tests

* Added some ST_Transform test functionality

* Additional transformation tests

* Somme additional transformation tests

* Print evaluated geom

* Done scaling geometry values

* Done scaling geometry values

* Improve collection coverage

* Remove printing

* Added ST_Buffer tests and new variant

* Removed new buffer variant

* Added coveralls badge

Co-authored-by: Idan Sheinberg <[email protected]>

* Done with aggregate testing

* Done with aggregate testing

* Formatting fixes

* Dremio compat fixes

* Lot's of work on buffer parsing

* Code now compiles

* Added ST_Buffer tests

* Added ST_Buffer tests

* Added ST_Buffer tests

* Coverage targets

* Version alignment

* Readme fix

Co-authored-by: Idan Sheinberg <[email protected]>
  • Loading branch information
sheinbergon and 4msheinbergon authored Oct 10, 2022
1 parent 1f52bad commit 685e789
Show file tree
Hide file tree
Showing 16 changed files with 782 additions and 57 deletions.
2 changes: 1 addition & 1 deletion .idea/kotlinc.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 7 additions & 7 deletions README.MD
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,13 @@

#### Version Compatbility

| Library Version | Dremio Version | Status |
|-----------------|----------------|------------------|
| 0.2.x | 20.1.x | Legacy |
| 0.3.x | 21.1.x | Maintained |
| 0.4.x | 21.2.x | Maintained |
| 0.5.x | 22.0.x | Maintained |
| 0.6.x | 22.1.x | Work In Progress |
| Library Version | Dremio Version | Status |
|-----------------|----------------|------------|
| 0.2.x | 20.1.x | Legacy |
| 0.3.x | 21.1.x | Maintained |
| 0.4.x | 21.2.x | Maintained |
| 0.5.x | 22.0.x | Maintained |
| 0.6.x | 22.1.x | Maintained |

### Usage Notes
As opposed to PostGIS, Dremio is only a query engine based on existing/projected data sources/lakes.
Expand Down
80 changes: 76 additions & 4 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,23 @@
<groupId>org.sheinbergon</groupId>
<artifactId>dremio-udf-gis</artifactId>
<properties>
<checkstyle.version>10.3</checkstyle.version>
<checkstyle.version>10.3.2</checkstyle.version>
<dremio.version>21.2.0-202205262146080444-038d6d1b</dremio.version>
<dremio-arrow.version>7.0.0-20220218200504-acc0d6e-dremio</dremio-arrow.version>
<proj4j.version>1.1.5</proj4j.version>
<jts-core.version>1.19.0</jts-core.version>
<jts-io-common.version>1.19.0</jts-io-common.version>
<commons-io.version>2.11.0</commons-io.version>
<junit.version>4.13.2</junit.version>
<junit-jupiter.version>5.9.0</junit-jupiter.version>
<kotlin.version>1.7.10</kotlin.version>
<kotlin.version>1.7.20</kotlin.version>
<kotest.version>5.4.2</kotest.version>
<joda.version>2.9</joda.version>
<detekt.version>1.21.0</detekt.version>
<carrotsearch.version>0.7.0</carrotsearch.version>
<arrow-memory-netty.version>8.0.0</arrow-memory-netty.version>
<arrow-memory-netty.version>7.0.0</arrow-memory-netty.version>
</properties>
<version>0.4.6${artifact.version.suffix}</version>
<version>0.4.8${artifact.version.suffix}</version>
<name>dremio-udf-gis</name>
<description>GIS UDF extensions for Dremio</description>
<url>https://github.com/sheinbergon/dremio-udf-gis</url>
Expand Down Expand Up @@ -148,6 +149,12 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>${commons-io.version}</version>
<scope>provided</scope>
</dependency>
<!-- Test Dependencies -->
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
Expand Down Expand Up @@ -399,6 +406,71 @@
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<version>${kotlin.version}</version>
<executions>
<execution>
<id>test-compile</id>
<phase>process-test-sources</phase>
<goals>
<goal>test-compile</goal>
</goals>
</execution>
<execution>
<id>compile</id>
<phase>process-sources</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
<configuration>
<jvmTarget>1.8</jvmTarget>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M7</version>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.8</version>
<executions>
<execution>
<id>prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.eluder.coveralls</groupId>
<artifactId>coveralls-maven-plugin</artifactId>
<version>4.3.0</version>
<configuration>
<sourceEncoding>UTF-8</sourceEncoding>
<repoToken>${env.COVERALLS_TOKEN}</repoToken>
</configuration>
<dependencies>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.3.1</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
<distributionManagement>
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/org/sheinbergon/dremio/udf/gis/STBuffer.java
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ public void setup() {

public void eval() {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
org.locationtech.jts.geom.Geometry buffered = geom.buffer(radiusInput.value);
org.locationtech.jts.geom.Geometry buffered = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.buffer(geom, radiusInput.value, null);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(buffered);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, binaryOutput);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
* <p>
* http://www.apache.org/licenses/LICENSE-2.0
* <p>
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.sheinbergon.dremio.udf.gis;

import com.dremio.exec.expr.SimpleFunction;
import com.dremio.exec.expr.annotations.FunctionTemplate;
import com.dremio.exec.expr.annotations.Output;
import com.dremio.exec.expr.annotations.Param;
import org.apache.arrow.memory.ArrowBuf;

import javax.inject.Inject;

@FunctionTemplate(
name = "ST_Buffer",
scope = FunctionTemplate.FunctionScope.SIMPLE,
nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
public class STBufferParameters implements SimpleFunction {
@Param
org.apache.arrow.vector.holders.NullableVarBinaryHolder binaryInput;

@Param(constant = true)
org.apache.arrow.vector.holders.Float8Holder radiusInput;

@Param
org.apache.arrow.vector.holders.NullableVarCharHolder parametersInput;

@Output
org.apache.arrow.vector.holders.NullableVarBinaryHolder binaryOutput;

@Inject
ArrowBuf buffer;

public void setup() {
}

public void eval() {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
java.lang.String parameters = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toUTF8String(parametersInput);
org.locationtech.jts.geom.Geometry buffered = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.buffer(geom, radiusInput.value, parameters);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(buffered);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, binaryOutput);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -42,38 +42,42 @@ public class STCollectAggregate implements AggrFunction {
org.apache.arrow.vector.holders.NullableVarBinaryHolder output;

@Inject
org.apache.arrow.memory.ArrowBuf buffer;
org.apache.arrow.memory.ArrowBuf valueBuffer;

@Inject
org.apache.arrow.memory.ArrowBuf outputBuffer;

@Override
public void setup() {
value = new org.apache.arrow.vector.holders.NullableVarBinaryHolder();
indicator = new org.apache.arrow.vector.holders.NullableBitHolder();
org.locationtech.jts.geom.GeometryCollection collection = org.sheinbergon.dremio.udf.gis.util.GeometryCollections.empty();
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(collection);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, value);
valueBuffer = valueBuffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, valueBuffer, value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderSet(value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(indicator);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.setValueFalse(indicator);
}

@Override
public void add() {
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isHolderSet(input)) {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(input);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(geom);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.append(bytes, buffer, value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderSet(indicator);
final int required = value.end + bytes.length + Integer.BYTES;
valueBuffer = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.enlargeBufferIfNeeded(valueBuffer, required);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.append(bytes, valueBuffer, value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.setValueTrue(indicator);
}
}

@Override
public void output() {
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isHolderSet(indicator)) {
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isValueTrue(indicator)) {
org.locationtech.jts.geom.GeometryCollection collection = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometryCollection(value);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(collection);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, output);
outputBuffer = outputBuffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, outputBuffer, output);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderSet(output);
} else {
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(output);
Expand All @@ -84,9 +88,9 @@ public void output() {
public void reset() {
org.locationtech.jts.geom.GeometryCollection collection = org.sheinbergon.dremio.udf.gis.util.GeometryCollections.empty();
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(collection);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, value);
valueBuffer = valueBuffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, valueBuffer, value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderSet(value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(indicator);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.setValueFalse(indicator);
}
}
29 changes: 16 additions & 13 deletions src/main/java/org/sheinbergon/dremio/udf/gis/STUnionAggregate.java
Original file line number Diff line number Diff line change
Expand Up @@ -42,18 +42,21 @@ public class STUnionAggregate implements AggrFunction {
org.apache.arrow.vector.holders.NullableVarBinaryHolder output;

@Inject
org.apache.arrow.memory.ArrowBuf buffer;
org.apache.arrow.memory.ArrowBuf valueBuffer;

@Inject
org.apache.arrow.memory.ArrowBuf outputBuffer;

@Override
public void setup() {
value = new org.apache.arrow.vector.holders.NullableVarBinaryHolder();
indicator = new org.apache.arrow.vector.holders.NullableBitHolder();
org.locationtech.jts.geom.Geometry empty = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.emptyGeometry();
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(empty);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, value);
valueBuffer = valueBuffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, valueBuffer, value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderSet(value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(indicator);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.setValueFalse(indicator);
}

@Override
Expand All @@ -63,19 +66,19 @@ public void add() {
org.locationtech.jts.geom.Geometry accumulated = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(value);
org.locationtech.jts.geom.Geometry union = accumulated.union(geom);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(union);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderSet(indicator);
valueBuffer = valueBuffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, valueBuffer, value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.setValueTrue(indicator);
}
}

@Override
public void output() {
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isHolderSet(indicator)) {
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isValueTrue(indicator)) {
org.locationtech.jts.geom.Geometry union = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(value);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(union);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, output);
outputBuffer = outputBuffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, outputBuffer, output);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderSet(output);
} else {
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(output);
Expand All @@ -86,9 +89,9 @@ public void output() {
public void reset() {
org.locationtech.jts.geom.Geometry empty = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.emptyGeometry();
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(empty);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, value);
valueBuffer = valueBuffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, valueBuffer, value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderSet(value);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(indicator);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.setValueFalse(indicator);
}
}
Loading

0 comments on commit 685e789

Please sign in to comment.