gRPC: Configurable proto input directory

[djnalluri]

Description
Currently, the input directory for compiling .proto files with the 'generate-code' goal is hard coded to 'src/main/proto'. It would be nice to make this configurable to a different directory. This is useful in use cases where the definitions are generated or downloaded at build time and hence, should be located in the target folder.

Implementation ideas
The generation code could read in a system property named "grpc.codegen.inputDir" to determine the input directory. It may use the existing behaviour if it isn't defined. Another option may be to provide a semicolon delimited list of directories to use in addition to src/main/proto for cases where only some of the definitions are dynamically provided or are spread out in multiple directories.

[quarkusbot]

/cc @michalszynkiewicz, @cescoffier

[michalszynkiewicz]

If we are going to do it, I think it should rather be a maven/gradle property, probably called: quarkus.generate-code.grpc.input-dir.

Could you write a bit more about your use case? Do you plan to use the downloaded proto files as sources for generation or includes?

[djnalluri]

We currently generate protobuf definitions via Maven plugin during each build. The definitions help define our gRPC services that map to a proprietary interface. So the definitions are not a source of truth as the interface may evolve underneath them. This plugin also allows us to limit generation to subsections of the entire interface depending on what is needed for a particular application so packaging all the definitions into a single unit is not optimal. We then use the protobuf-maven-plugin to generate Java clients. This last step is what we would like to simplify to use just the generate-code goal in the quarkus-maven-plugin.

[michalszynkiewicz]

@djnalluri I finally have some time for it and I'm rereading our conversation.
There is one thing that cought my eye, you wrote:
So the definitions are not a source of truth as the interface may evolve underneath them

Do you mean that you generate the Java files from proto once and then modify Java files?

[djnalluri]

Apologies for the poor wording. We use a proprietary software/framework for developing a legacy application. We have a Maven plugin connect to an active instance and scan some internal schemas to determine what interactions are actually possible. The intention is to wrap these interactions with gRPC. So what I meant by that statement was that we don't know for sure what gRPC methods may be supported until build time when we actually analyze the instance. We don't do any hand editing of the generated files. The legacy code follows a separate release/dev cycle and has a different team. We do have some custom generated gRPC server code to handle the conversion but our primary use case is to streamline building the client side (no customization required) as it's used in multiple downstream builds. This may have been partially solved by the importing of proto files from dependencies added in 1.10 btw so thanks for that 👍 .

[michalszynkiewicz]

@djnalluri do you know now if importing proto files solves the problem for you?

[djnalluri]

I haven't been able to prioritize building this out. It would be a more involved setup as we would have to create an artifact with versions for each available instance. Currently, there's no way to look up the available instances and their URL so we would have to hard code them into the CI.

[philippegoncalves]

@michalszynkiewicz for gradle, why not using something similar to https://github.com/google/protobuf-gradle-plugin:
sourceSets { main { proto { // In addition to the default 'src/main/proto' srcDir 'src/main/protobuf' srcDir 'src/main/protocolbuffers' // In addition to the default '**/*.proto' (use with caution). // Using an extension other than 'proto' is NOT recommended, // because when proto files are published along with class files, we can // only tell the type of a file from its extension. include '**/*.protodevel' } java { ... } } test { proto { // In addition to the default 'src/test/proto' srcDir 'src/test/protocolbuffers' } } }

Source folders should be defined in the sourceSets block, similar to what different gradle plugins do. In the android world for example, gradle allows to redefine not only the java input source folder (supported by gradle by default), but almost all the android code components (manifest file, assets, etc).
We have the same use case currently in our project, where the proto files dont live inside the code, but are fetched part of the configuration phase.

[michalszynkiewicz]

@philippegoncalves thanks for your input.

BTW, you can use the regular gradle plugin with your quarkus app, just remember to add quarkus-grpc-protoc-plugin, similarly to https://quarkus.io/guides/grpc-getting-started#generating-java-files-from-proto-with-protobuf-maven-plugin

[philippegoncalves]

@michalszynkiewicz problem using com.google.protobuf is that io.quarkus and com.google.protobuf gradle plugins are incompatible when using proto files, because both will generate the stubs which will create a duplicate class issue at compile time.
A workaround for this is to redefine the proto folder to not point to src/main/proto, this way the quarkus plugin wont grab the proto files. But then we need to tell protoc which plugin to use to generate the files (quarkus-grpc-protoc-plugin) but it is expecting an artifact with an execution binary (which will depends on the architecture used to compile the project).

So far our approach for this issue, using io.quarkus is:

• define a new configuration for our internal proto files, like

configurations {
    myProto
}

• Use this configuration for the dependency

dependencies {
...
    myProto 'com.xxx.yyy:helloworld:1.0.0'
}

• Use a custom task that copy the proto files to the src/main/proto/myProtos folder and add a dependency to compilation:

task copyCustomProto {
    doLast {
        copy{
            from zipTree(configurations.myProto.singleFile)
            into "src/main/proto/myProtos"
        }
    }
}

tasks.getByName("compileJava").dependsOn("copyCustomProto")

• add src/main/proto/myProtos to .gitignore files

[michalszynkiewicz]

You can skip the grpc code generation that is built into Quarkus by setting grpc.codegen.skip system property to true

[michalszynkiewicz]

FYI, now it is possible to generate proto stubs from dependencies, that is jars contaning proto files.

[drsarutobi8]

Hi @michalszynkiewicz ,
I auto generate protos from my CRUD services and I write business protos separately. So I agree with @philippegoncalves that having quarkus to code generate from sourceSet/main/proto where I can put reference to both types of proto there together (without com.google.protobuf) would be cleaner.

[michalszynkiewicz]

@drsarutobi8 how does your app get the protos?

[drsarutobi8]

@michalszynkiewicz
Sorry, it was over a year.
I am working on the project that automatically generates Quarkus entity classes, CRUD service classes, proto files and many others from database definition language. This lessens mundane works and leave only essential works for developer. So all files would be under build/gensrc/xx.

However, developer would want to put his own custom classes, methods, and protos. It would be nice that he/she can put them under src/main/protos and then using srcSet to point to src/main/protos and build/gensrc/protos.

Br.

[LarsSven]

Is there any update on this?

Having the ability to configure the source folder for the proto files would be extremely useful to our project.