Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[FEAT] add spark-connect protocol (#3189)
This PR adds a script-based approach for generating the Spark Connect protobuf code, eliminating the need for developers to have the protobuf compiler (protoc) installed locally. Key changes: - Added a new `spark-connect-script` crate that uses `tonic-build` to compile the .proto files into Rust code at build time - The script outputs the generated code as a string which can then be stored directly in the codebase - Added the generated Rust code to `src/spark-connect/src/lib.rs` - Updated workspace members to include both new crates - The proto files are copied from Spark's codebase and cover the full Connect protocol including base, catalog, commands, common, expressions, relations and types This approach makes it easier for developers to work with the Spark Connect protocol since they don't need to manage protoc installation and versioning. The generated code is checked into the repo and can be reviewed/updated as needed. Dependencies added: - prost: For protobuf encoding/decoding - prost-types: Common protobuf type definitions - tonic: For gRPC functionality
- Loading branch information