Tritonserver bindings without GPU

[jbkyang-nvi]

Created this branch to ask about CPU only build difficulties.

A customer needs to run Triton on a machine without GPU so I tried removing the cuda + TensorRT dependencies
in the Triton bindings.
I followed the following steps:

 $ docker run -it --gpus=all -v $(pwd):/workspace nvcr.io/nvidia/tritonserver:22.01-py3 bash
 $ apt update
 $ apt install -y openjdk-11-jdk
 $ wget https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz
 $ tar zxvf apache-maven-3.8.4-bin.tar.gz
 $ export PATH=/opt/tritonserver/apache-maven-3.8.4/bin:$PATH
 $ git clone https://github.com/jbkyang-nvi/javacpp-presets/tree/kyang-cpu-only
 $ cd javacpp-presets
 $ mvn clean install --projects .,tritonserver
 $ mvn clean install -f platform --projects ../tritonserver/platform -Djavacpp.platform=linux-x86_64

Everything built ok with the exception of a few warnings.
However, when I tried the build the SimpleCPUOnly.java project with

 $ cd tritonserver/samples
 $ mvn compile exec:java -Djavacpp.platform=linux-x86_64 -Dexec.args="-r /workspace/models"

I got compilation errors. There seemed to be issues converting from java String and BytePointer and boolean[] to BoolPointer. This is quite strange as this is a supported type and the GPU enabled version compiled fine.

@saudet can you advise on what I might be doing incorrectly?

Thanks
The full error log:

root@9186e52a3082:/opt/tritonserver/javacpp-presets/tritonserver/samples# mvn clean install -Djavacpp.platform=linux-x86_64
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------< org.bytedeco.tritonserver:simple >------------------
[INFO] Building simple 1.5.7-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ simple ---
[INFO] Deleting /opt/tritonserver/javacpp-presets/tritonserver/samples/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ simple ---
[WARNING] Using platform encoding (ANSI_X3.4-1968 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /opt/tritonserver/javacpp-presets/tritonserver/samples/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ simple ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding ANSI_X3.4-1968, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /opt/tritonserver/javacpp-presets/tritonserver/samples/target/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING :
[INFO] -------------------------------------------------------------
[WARNING] /opt/tritonserver/javacpp-presets/tritonserver/samples/Simple.java: /opt/tritonserver/javacpp-presets/tritonserver/samples/Simple.java uses or overrides a deprecated API.
[WARNING] /opt/tritonserver/javacpp-presets/tritonserver/samples/Simple.java: Recompile with -Xlint:deprecation for details.
[INFO] 2 warnings
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /opt/tritonserver/javacpp-presets/tritonserver/samples/Simple.java:[84,9] method does not override or implement a method from a supertype
[ERROR] /opt/tritonserver/javacpp-presets/tritonserver/samples/Simple.java:[513,13] no suitable method found for TRITONSERVER_ServerModelIsReady(Simple.TRITONSERVER_ServerDeleter,java.lang.String,int,boolean[])
    method org.bytedeco.tritonserver.global.tritonserver.TRITONSERVER_ServerModelIsReady(org.bytedeco.tritonserver.tritonserver.TRITONSERVER_Server,java.lang.String,long,org.bytedeco.javacpp.BoolPointer) is not applicable
      (argument mismatch; boolean[] cannot be converted to org.bytedeco.javacpp.BoolPointer)
    method org.bytedeco.tritonserver.global.tritonserver.TRITONSERVER_ServerModelIsReady(org.bytedeco.tritonserver.tritonserver.TRITONSERVER_Server,org.bytedeco.javacpp.BytePointer,long,boolean[]) is not applicable
      (argument mismatch; java.lang.String cannot be converted to org.bytedeco.javacpp.BytePointer)
[INFO] 2 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.130 s
[INFO] Finished at: 2022-02-08T00:41:32Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project simple: Compilation failure: Compilation failure:
[ERROR] /opt/tritonserver/javacpp-presets/tritonserver/samples/Simple.java:[84,9] method does not override or implement a method from a supertype
[ERROR] /opt/tritonserver/javacpp-presets/tritonserver/samples/Simple.java:[513,13] no suitable method found for TRITONSERVER_ServerModelIsReady(Simple.TRITONSERVER_ServerDeleter,java.lang.String,int,boolean[])
[ERROR]     method org.bytedeco.tritonserver.global.tritonserver.TRITONSERVER_ServerModelIsReady(org.bytedeco.tritonserver.tritonserver.TRITONSERVER_Server,java.lang.String,long,org.bytedeco.javacpp.BoolPointer) is not applicable
[ERROR]       (argument mismatch; boolean[] cannot be converted to org.bytedeco.javacpp.BoolPointer)
[ERROR]     method org.bytedeco.tritonserver.global.tritonserver.TRITONSERVER_ServerModelIsReady(org.bytedeco.tritonserver.tritonserver.TRITONSERVER_Server,org.bytedeco.javacpp.BytePointer,long,boolean[]) is not applicable
[ERROR]       (argument mismatch; java.lang.String cannot be converted to org.bytedeco.javacpp.BytePointer)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[saudet]

There's most likely a few lines of InfoMap that get inherited from CUDA and/or TensorRT, probably at least this one that you can add to presets/tritonserver.java, but maybe others too:
https://github.com/bytedeco/javacpp-presets/blob/master/tensorrt/src/main/java/org/bytedeco/tensorrt/presets/nvinfer.java#L121
In any case, when the files in tritonserver/src/gen/ remain unchanged, you'll know you have them all.

[jbkyang-nvi]

That worked! Thanks :)

[saudet]

BTW, if Triton itself doesn't depend on CUDA, we can probably merge these changes and it will still work with CUDA, as long as Triton can load the libraries for CUDA by itself.

[jbkyang-nvi]

Agreed. We're working on a solution in Tritonserver client (where we would want to keep the presets so we can maintain a monthly release cycle). Will do a big migration anyway so we can just wait for that for now.