Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create prototype gRPC getCompactionJob() service for external compactions #4715

Closed
wants to merge 35 commits into from
Closed
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
212ca11
Add grpc and protobuf setup to maven build
cshannon Jun 30, 2024
75eb46f
Add defintion and generate service/protocol for compaction service
cshannon Jun 30, 2024
00fcedd
Update the Compactor and Compaction coordinator to use new grpc service
cshannon Jun 30, 2024
1234d67
Clean up comments from example
cshannon Jun 30, 2024
0dc1df8
clean up comments in compaction-coordinator.proto
cshannon Jul 1, 2024
450d371
Add generated protobuf to .gitattributes
cshannon Jul 1, 2024
d7dd27d
Merge branch 'elasticity' into accumulo-4664-grpc
cshannon Jul 5, 2024
f27d6c5
Merge branch 'elasticity' into accumulo-4664-grpc
cshannon Jul 5, 2024
17e7aa4
Switch gRPC getCompactionJob() to using new async method
cshannon Jul 5, 2024
edf7125
Use thenAccept() as we don't need a return value
cshannon Jul 5, 2024
dae30cf
fix errorprone warning and send back exception on error to client
cshannon Jul 5, 2024
36481c7
Merge branch 'elasticity' into accumulo-4664-grpc
cshannon Jul 12, 2024
4e72058
Move protobuf objects into correct packages
cshannon Jul 12, 2024
3fb40bd
Add license headers to generated protobuf
cshannon Jul 12, 2024
c5c09af
Move compactionCompleted to gRPC, share grpc connection
cshannon Jul 13, 2024
854ee44
Move more CompactionService rpc calls to gRpc
cshannon Jul 13, 2024
23369fb
Merge branch 'elasticity' into accumulo-4664-grpc
cshannon Jul 19, 2024
548d2ca
Move new gRPC generation to a new module
cshannon Jul 19, 2024
80e5cb2
fix build with relative path in pom
cshannon Jul 19, 2024
3efd1e9
update .gitattributes after moving grpc
cshannon Jul 19, 2024
d57af85
address comments
cshannon Jul 19, 2024
c7b7e8f
Improve rpc creds in ClientContext and switch to string for PKeyExten…
cshannon Jul 19, 2024
3fc6be0
add a property for grpc port
cshannon Jul 19, 2024
9da244a
Move rest of compaction coordinator service to gRPC and update Monitor
cshannon Jul 20, 2024
f000c46
Remove no longer needed Thrift compaction coordinator service from Co…
cshannon Jul 20, 2024
a1b53ad
Merge branch 'elasticity' into accumulo-4664-grpc
cshannon Jul 20, 2024
936edd1
QA build fixes
cshannon Jul 20, 2024
fc8411f
fix test
cshannon Jul 21, 2024
4a02a28
Merge branch 'grpc' into accumulo-4664-grpc
cshannon Jul 26, 2024
a93f989
Merge branch 'grpc' into accumulo-4664-grpc
cshannon Aug 9, 2024
cd3573c
Merge branch 'grpc' into accumulo-4664-grpc
cshannon Aug 10, 2024
9b1c77f
Merge branch 'grpc' into accumulo-4664-grpc
cshannon Aug 16, 2024
66cb2cf
Add initial support for TLS for server/client
cshannon Aug 17, 2024
fa3a8b6
fix formatting
cshannon Aug 17, 2024
32f09d1
Merge branch 'grpc' into accumulo-4664-grpc
cshannon Aug 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,10 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
Expand All @@ -69,6 +73,14 @@
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
Expand Down Expand Up @@ -143,6 +155,17 @@
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
</dependency>
<dependency>
<!-- necessary for protobuf and Java 9+ -->
<groupId>org.apache.tomcat</groupId>
<artifactId>annotations-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty-shaded</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-runtime</artifactId>
Expand Down Expand Up @@ -258,12 +281,14 @@
<excludes>
<exclude>.*[.]impl[.].*</exclude>
<exclude>.*[.]thrift[.].*</exclude>
<exclude>.*[.]protobuf[.].*</exclude>
<exclude>org[.]apache[.]accumulo[.]core[.]security[.]crypto[.].*</exclude>
</excludes>
<allows>
<!--Allow API data types to reference thrift types, but do not
analyze thrift types -->
<allow>org[.]apache[.]accumulo[.].*[.]thrift[.].*</allow>
<allow>org[.]apache[.]accumulo[.].*[.]protobuf[.].*</allow>
<!--Type from hadoop used in API. If adding a new type from
Hadoop to the Accumulo API ensure its annotated as
stable.-->
Expand Down Expand Up @@ -393,6 +418,13 @@
</executions>
</plugin>
</plugins>
<extensions>
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>1.7.1</version>
</extension>
</extensions>
</build>
<profiles>
<profile>
Expand All @@ -418,6 +450,43 @@
</plugins>
</build>
</profile>
<profile>
<id>protobuf</id>
<build>
<plugins>
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.6.1</version>
<configuration>
<protocArtifact>com.google.protobuf:protoc:3.25.3:exe:${os.detected.classifier}</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:1.64.0:exe:${os.detected.classifier}</pluginArtifact>
<outputDirectory>src/main/protobuf-gen-java/</outputDirectory>
</configuration>
<executions>
<execution>
<id>generate-protobuf</id>
<goals>
<goal>compile</goal>
</goals>
<phase>generate-sources</phase>
</execution>
<execution>
<id>generate-grpc</id>
<goals>
<goal>compile-custom</goal>
</goals>
<phase>generate-sources</phase>
<configuration>
<clearOutputDirectory>false</clearOutputDirectory>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<!-- profile for running second execution of surefire without reusing forks -->
<!-- on by default, but disable with '-P !reuseForksFalse' or '-Dtest=SomeTest' -->
Expand Down
Loading