[Kernel] Add examples for Delta Kernel API usage #1926

vkorukanti · 2023-07-21T09:02:45Z

Description

Adds an example project that shows how to read a Delta table using the Kernel APIs. The sample program can also be used as a command line to read the Delta table.

Single threaded reader

java io.delta.kernel.examples.SingleThreadedTableReader \
    --table=file:<repo-dir>/connectors/golden-tables/src/main/resources/golden/data-reader-primitives \
    --columns=as_int,as_long
    --limit=5

              as_int|             as_long
                null|                null
                   0|                   0
                   1|                   1
                   2|                   2
                   3|                   3

Multi-threaded reader (simulating a distributed execution environment)

java io.delta.kernel.examples.MultiThreadedTableReader
    --table=file:<repo-dir>/connectors/golden-tables/src/main/resources/golden/data-reader-primitives \
    --columns=as_int,as_long
    --limit=20
    --parallelism=5

              as_int|             as_long
                null|                null
                   0|                   0
                   1|                   1
                   2|                   2
                   3|                   3

How was this patch tested?

Manual testing

Usage: java io.delta.kernel.examples.SingleThreadedTableReader [-c <arg>] [-l <arg>] -t <arg>
   -c,--columns <arg>   Comma separated list of columns to read from the table. Ex. --columns=id,name,address
   -l,--limit <arg>     Maximum number of rows to read from the table (default 20).
  -t,--table <arg>     Fully qualified table path

 Usage: java io.delta.kernel.examples.MultiThreadedTableReader [-c <arg>] [-l <arg>] [-p <arg>] -t <arg>
    -c,--columns <arg>       Comma separated list of columns to read from the table. Ex. --columns=id,name,address
    -l,--limit <arg>         Maximum number of rows to read from the table (default 20).
    -p,--parallelism <arg>   Number of parallel readers to use (default 3).
    -t,--table <arg>         Fully qualified table path

tdas · 2023-08-01T17:45:38Z

kernel/examples/table-reader/pom.xml

+
+    <groupId>org.example</groupId>
+    <artifactId>table-reader</artifactId>
+    <version>3.0.0-SNAPSHOT</version>


should we be hardcoding this? also why does this need to be same version as the actual delta artifacts that we will publish. unless there is a need to consistent, this is only going to cause more headache to keep it updated?

Updated the version to 0.1-SNAPSHOT. Let me know if there is a better version name.

tdas · 2023-08-01T17:46:36Z

kernel/examples/run_examples.sh

@@ -0,0 +1,49 @@
+##


tdas · 2023-08-01T17:47:09Z

kernel/examples/run_examples.sh

+## tables located in the <repo-root>/connectors/golden-tables/src/main/resources/golden
+## directory.
+##
+## Make sure to run this script from <repo-root> in order for the relative


why not make this script be runnable from anywhere?

this sort of lack of flexibility causes future pain.

also .. thats why writing such scripts in python gives a lot more flexibility and an entire library of tools to do these sort of stuff. There is a lot of python scripts in this repo (e.g., run-integration-tests.py) that you can steal from.

Rewrote this in python (by copying the code from delta integration test).

kernel/examples/run_examples.sh

tdas · 2023-08-01T17:50:32Z

kernel/examples/run_examples.sh

+    -Dstaging.repo.url=${EXTRA_MAVEN_REPO:-"___"} \
+    -Ddelta-kernel.version=${STANDALONE_VERSION:-"3.0.0-SNAPSHOT"} \
+    -Dexec.args="${test}"
+done


have you tested that this scripts fails (exit nonzero) when compilation and test in the examples fail?

tdas · 2023-08-01T17:55:37Z

.../examples/table-reader/src/main/java/io/delta/kernel/examples/SingleThreadedTableReader.java

+        StructType readSchema,
+        Snapshot snapshot,
+        int maxRowCount) throws IOException
+    {


we are mixing styles all over the place.. even within this function. can we stick to the following since we are all familiar with it (consistent with all the scala code).

try { ... }

This follows our Java checkstyle guide and Kernel Java code is. Let me know if this needs to be changed.

tdas · 2023-08-01T17:56:10Z

kernel/examples/table-reader/src/main/java/io/delta/kernel/examples/utils/RowSerDe.java

+import io.delta.kernel.data.DefaultJsonRow;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.internal.types.TableSchemaSerDe;
+import io.delta.kernel.types.ArrayType;


nit: cant this be just types.*

Again, this is following the Java checkstyle. Also using wildcard in imports is not recommended.

tdas · 2023-08-01T17:57:15Z

.../examples/table-reader/src/main/java/io/delta/kernel/examples/SingleThreadedTableReader.java

+     * @return
+     * @throws Exception
+     */
+    private void readSnapshot(


readData

snapshot is ambiguous. snapshot metadata or snapshot data?

ok, changed to readData

...l/examples/table-reader/src/main/java/io/delta/kernel/examples/MultiThreadedTableReader.java

kernel/examples/run-kernel-examples.py

tdas · 2023-08-02T16:02:15Z

approved, but I left some comments.

vkorukanti force-pushed the kernel-examples branch 2 times, most recently from d59897b to 1d0c173 Compare July 21, 2023 17:27

vkorukanti force-pushed the kernel-examples branch from 5bab08b to 92b4189 Compare July 28, 2023 17:22

vkorukanti changed the title ~~[Kernel][WIP] Add examples for Delta Kernel API usage~~ [Kernel] Add examples for Delta Kernel API usage Jul 28, 2023

vkorukanti requested review from tdas and allisonport-db July 28, 2023 17:26

Add examples for Delta Kernel API usage

5f9b4e2

vkorukanti force-pushed the kernel-examples branch from 92b4189 to 5f9b4e2 Compare July 31, 2023 17:17

tdas reviewed Aug 1, 2023

View reviewed changes

kernel/examples/run_examples.sh Outdated

@@ -0,0 +1,49 @@

##

Copy link

Contributor

tdas Aug 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

license.

tdas reviewed Aug 1, 2023

View reviewed changes

kernel/examples/run_examples.sh Outdated Show resolved Hide resolved

tdas reviewed Aug 1, 2023

View reviewed changes

Anonymous added 2 commits August 1, 2023 14:17

update the script to python

a55aca0

f

bfedfb6

tdas reviewed Aug 2, 2023

View reviewed changes

...l/examples/table-reader/src/main/java/io/delta/kernel/examples/MultiThreadedTableReader.java Outdated Show resolved Hide resolved

tdas reviewed Aug 2, 2023

View reviewed changes

kernel/examples/run-kernel-examples.py Outdated Show resolved Hide resolved

tdas reviewed Aug 2, 2023

View reviewed changes

kernel/examples/run-kernel-examples.py Outdated Show resolved Hide resolved

tdas approved these changes Aug 2, 2023

View reviewed changes

review

9b91978

vkorukanti merged commit 278e1d2 into delta-io:master Aug 2, 2023

vkorukanti deleted the kernel-examples branch September 14, 2023 11:55

vkorukanti mentioned this pull request Sep 28, 2023

[Umbrella Feature Request] Delta Kernel APIs to simplify building connectors for reading Delta tables #1783

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Kernel] Add examples for Delta Kernel API usage #1926

[Kernel] Add examples for Delta Kernel API usage #1926

vkorukanti commented Jul 21, 2023 •

edited

Loading

tdas Aug 1, 2023

vkorukanti Aug 1, 2023

tdas Aug 1, 2023

tdas Aug 1, 2023

tdas Aug 1, 2023

tdas Aug 1, 2023

vkorukanti Aug 1, 2023

tdas Aug 1, 2023 •

edited

Loading

tdas Aug 1, 2023

vkorukanti Aug 1, 2023

tdas Aug 1, 2023

vkorukanti Aug 1, 2023

tdas Aug 1, 2023

vkorukanti Aug 1, 2023

tdas commented Aug 2, 2023

[Kernel] Add examples for Delta Kernel API usage #1926

[Kernel] Add examples for Delta Kernel API usage #1926

Conversation

vkorukanti commented Jul 21, 2023 • edited Loading

Description

How was this patch tested?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdas Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdas commented Aug 2, 2023

vkorukanti commented Jul 21, 2023 •

edited

Loading

tdas Aug 1, 2023 •

edited

Loading