Introduce a Kubernetes client

[jbrisbin]

In an attempt to deploy a test Operator into Kubernetes using QuarkusIO, I encountered errors using both the Fabric8 client and the official Java client. Fabric8 uses reflection in some base classes, so fails at runtime and the official Java client had errors compiling to a native image because of optional dependencies for Azure Kubernetes.

It would be really awesome to have native-image support for Kubernetes Operators via a client that was developed for (and tested on) AWS, Azure, GKE, minikube, OpenShift, etc... so that Kubernetes users could easily on-ramp resources that connect to the API server (a la Operators).

[geoand]

cc @iocanel @rhuss @Jiri-Kremser

[geoand]

As far as I know @iocanel is working on eliminating the reflection from the KubernetesClient. Once that is done, I think we could utilize Quarkus in the jvm-operators project that @Jiri-Kremser has created

[iocanel]

Right. Here's the relevant pull request: fabric8io/kubernetes-client#1433

[jbrisbin]

I'd love to try this out again once this is merged and available in central.

[jkremser]

As far as I know @iocanel is working on eliminating the reflection from the KubernetesClient. Once that is done, I think we could utilize Quarkus in the jvm-operators project that @Jiri-Kremser has created

Definitely, the maven plugin for the native builds looks promising. Some changes needs to be done to the current abstract-operator, like removing the reflection calls (or putting them in the list to that json file) and some minor refactoring of the static methods, but nothing difficult. Currently, I am blocked by the run-time issues in the f8-k8s client when performing calls using the ok http client.

[cescoffier]

@iocanel @geoand A quickstart or blog would be very very nice!

[rhuss]

@iocanel Do you think that the kubernetes-client will be able to run ootb with Quarkus or do you think we would need a dedicated extension (like mention in fabric8io/kubernetes-client#1427) ?

[iocanel]

@rhuss: I think it eventually will be able to run out of the box. I don't see why we would need an extension.

[ricardozanini]

For whom might be interested, I made some experimentations with both k8s clients (Fabric8 and the Client Java) in native environment that can be viewed here.

Basically both worked pretty well with token authentication relying on registering the domain model to be used with reflection. In a more broader usage scenario, the entire domain model should be registered.

[geoand]

@ricardozanini Nice! Could you describe what changes you needed to make? We haven't actually done any real work on our side, but the very simple attempts we had done to use the kubernetes client with quarkus to create a native binary stumbled on some okhttp + ssl issues on graalvm

[ruromero]

FWIW I did few tests today with kubernetes client and I just had to enable the JNI flag to make the native build do a simple watch.
Monstly what is explained in the infinispan guide https://quarkus.io/guides/infinispan-client-guide

[geoand]

Very interesting, @iocanel and I will have to take a closer look

[ricardozanini]

Hi @geoand!

For the "official" kubernetes client I had to:

1. Add --allow-incomplete-classpath because of some incompatibility with Azure libraries. I haven't dig so far to understand why, since I won't use Azure at this stage.
2. Enable security services and JNI to be able to communicate with HTTPS protocol.
3. Manually register the domain model that I'm using to query the Service API.
4. Manually set the java.library.path and javax.net.ssl.trustStore properties since Quarkus wasn't injecting them automatically (even though I have the GRAALVM_HOME correctly set in my env). I haven't the time to investigate it yet.

For the Fabric8 client, was almost the same configuration except for the --allow-incomplete-classpath build argument.

The problem lies on the JSON Serialization library. Kubernetes Java uses GSON and Fabiric8 Jackson. Both heavy reflection users. I intent to have a look on how you guys are working with JSON serialization with JSON-B to implement our own k8s client for our needs (at least at this stage of development), since we are trying to stay away from reflection at all. The last resort will rely on JSON Path libraries to get what we need from the API responses. :(

[geoand]

Thanks for the info!

[emmanuelbernard]

Nice @ricardozanini !
So --allow-incomplete-classpath is something we are wrestling with in our GraalVM 19 support. So let me CC @dmlloyd here.
We do support jackson (for RESTEasy at least) in Quarkus. So that might work for the Fabric8 part. If you know the JSON types ahead of time, then a kubernetes-(fabric8-)client extension can very easily enlist them.

[geoand]

There is now a PR for this.

[jkremser]

this was my "one-liner" :)

docker kill `docker ps -q` || true && mvn clean install && cp /home/jkremser/install/graalvm-ce-1.0.0-rc16/jre/lib/amd64/libsunec.so ./target/ && ~/install/graalvm-ce-1.0.0-rc13/bin/native-image --no-server -H:Name=micro -J-Xmx4G -J-Xms4G -H:+JNI -H:EnableURLProtocols=https --allow-incomplete-classpath -H:ReflectionConfigurationFiles=classes.json -H:+ReportExceptionStackTraces -H:+ReportUnsupportedElementsAtRuntime --static -H:Name=app -jar ./target/spark-operator-*.jar && mv ./spark-operator-0.3.6-SNAPSHOT target/ && docker build -f Dockerfile.centos . -t foo && oc cluster up && oc login -u system:admin ; oc project default && sleep 2 && oc apply -f manifest/operator.yaml && sleep 8 && opLogs

[geoand]

@jkremser The PR assuming it will be merged will shield you from all that GraalVM native-image complexity :)

[lordofthejars]

PR for not using reflection has been released in the kubernetes client since 4.2.0 currently we are 4.3.0 so it should work.

[geoand]

@lordofthejars there were still some reflection issues that needed to be addressed to make the Client work on GraalVM, hence the need for the extension

[lordofthejars]

Sure, anything you need from Kubernetes client just pings us and we will release ASAP.

[geoand]

Thanks :) We should be good for the time being, but I'll definitely let you know whatever comes up