Skip to content

Commit

Permalink
[FEAT] add spark-connect protocol (#3189)
Browse files Browse the repository at this point in the history
This PR adds a script-based approach for generating the Spark Connect
protobuf code, eliminating the need for developers to have the protobuf
compiler (protoc) installed locally.

Key changes:

- Added a new `spark-connect-script` crate that uses `tonic-build` to
compile the .proto files into Rust code at build time
- The script outputs the generated code as a string which can then be
stored directly in the codebase
- Added the generated Rust code to `src/spark-connect/src/lib.rs`
- Updated workspace members to include both new crates
- The proto files are copied from Spark's codebase and cover the full
Connect protocol including base, catalog, commands, common, expressions,
relations and types

This approach makes it easier for developers to work with the Spark
Connect protocol since they don't need to manage protoc installation and
versioning. The generated code is checked into the repo and can be
reviewed/updated as needed.

Dependencies added:
- prost: For protobuf encoding/decoding
- prost-types: Common protobuf type definitions  
- tonic: For gRPC functionality
  • Loading branch information
andrewgazelka authored Nov 8, 2024
1 parent bdd25b6 commit f566125
Show file tree
Hide file tree
Showing 16 changed files with 10,355 additions and 68 deletions.
380 changes: 313 additions & 67 deletions Cargo.lock

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,9 @@ members = [
"src/daft-table",
"src/daft-writers",
"src/hyperloglog",
"src/parquet2"
"src/parquet2",
# "src/spark-connect-script",
"src/generated/spark-connect"
]

[workspace.dependencies]
Expand Down
12 changes: 12 additions & 0 deletions src/generated/spark-connect/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
[dependencies]
prost = "0.13.3"
prost-types = "0.13.3"
tonic = "0.12.3"

[lints]
workspace = true

[package]
edition = {workspace = true}
name = "spark-connect"
version = {workspace = true}
Loading

0 comments on commit f566125

Please sign in to comment.