Hacking on OAuth and deploying with Strimzi Kafka Operator

.travis/build.sh

@@ -26,6 +26,7 @@ mvn spotbugs:check
 
 # Run testsuite with java 8 only
 if [ ${JAVA_MAJOR_VERSION} -eq 1 ] ; then
+  docker pull oryd/hydra:v1.0.0
   mvn test-compile spotbugs:check -e -V -B -f testsuite
   set +e
   mvn -e -V -B install -f testsuite

RELEASE_NOTES.md

@@ -0,0 +1,71 @@
+Release Notes
+=============
+
+0.4.0
+-----
+
+### Deprecated configuration options
+
+The following configuration options have been deprecated:
+* `oauth.tokens.not.jwt` is now called `oauth.access.token.is.jwt` and has a reverse meaning.
+* `oauth.validation.skip.type.check` is now called `oauth.check.access.token.type` and has a reverse meaning.
+
+
+See: Align configuration with Kafka Operator PR ([#36](https://github.com/strimzi/strimzi-kafka-oauth/pull/36)).
+
+### Compatibility improvements
+
+Scope claim is no longer required in an access token. ([#30](https://github.com/strimzi/strimzi-kafka-oauth/pull/30))
+That improves compatibility with different authorization servers, since the attribute is not required by OAuth 2.0 specification neither is it used by validation logic.
+
+### Updated dependencies
+
+`jackson-core`, and `jackson-databind` libraries have been updated to latest versions. ([#33](https://github.com/strimzi/strimzi-kafka-oauth/pull/33))
+
+### Instructions for developers added
+
+Instructions for preparing the environment, building and deploying the latest version of Strimzi Kafka OAuth library with Strimzi Kafka Operator have been added.
+
+See: Hacking on OAuth and deploying with Strimzi Kafka Operator PR ([#34](https://github.com/strimzi/strimzi-kafka-oauth/pull/34))
+
+### Improvements to examples and documentation
+
+Fixed enabled remote debugging mode in example `compose-authz.yml` ([#36](https://github.com/strimzi/strimzi-kafka-oauth/pull/36))
+
+0.3.0
+-----
+
+### Token-based authorization with Keycloak Authorization Services
+
+It is now possible to use Keycloak Authorization Services to centrally manage access control to resources on Kafka Brokers ([#36](https://github.com/strimzi/strimzi-kafka-oauth/pull/36))
+See the [tutorial](examples/README-authz.md) which explains many concepts.
+For configuration details also see [KeycloakRBACAuthorizer JavaDoc](oauth-keycloak-authorizer/src/main/java/io/strimzi/kafka/oauth/server/authorizer/KeycloakRBACAuthorizer.java). 
+
+### ECDSA signature verification support
+
+The JWTSignatureValidator now supports ECDSA signatures, but requires explicit enablement of BouncyCastle security provider ([#25](https://github.com/strimzi/strimzi-kafka-oauth/pull/25))
+To enable BouncyCastle set `oauth.crypto.provider.bouncycastle` to `true`.
+Optionally you may control the order where the provider is installed by using `oauth.crypto.provider.bouncycastle.position` - by default it is installed at the end of the list of existing providers.
+
+0.2.0
+-----
+
+### Testsuite with integration tests
+
+A testsuite based on Arquillian Cube, and using docker containers was added.
+
+### Examples improvements
+
+Added Ory Hydra authorization server to examples. 
+
+0.1.0
+-----
+
+### Initial OAuth 2 authentication support for Kafka
+
+Support for token-based authentication that plugs into Kafka's SASL_OAUTHBEARER mechanism to provide:
+* Different ways of access token retrieval for Kafka clients (clientId + secret, refresh token, or direct access token)
+* Fast signature-checking token validation mechanism (using authorization server's JWKS endpoint)
+* Introspection based token validation mechanism (using authorization server's introspection endpoint)
+
+See the [tutorial](examples/README.md).

examples/docker/strimzi-kafka-image/Dockerfile

@@ -0,0 +1,4 @@
+FROM strimzi/kafka:latest-kafka-2.4.0
+
+COPY target/libs/* /opt/kafka/libs/oauth/
+ENV CLASSPATH /opt/kafka/libs/oauth/*

examples/docker/strimzi-kafka-image/README.md

@@ -0,0 +1,97 @@
+Strimzi Kafka Image with SNAPSHOT Strimzi Kafka OAuth
+=====================================================
+
+This is a build of a Docker image based on `strimzi/kafka:latest-kafka-2.4.0` with added most recently locally built SNAPSHOT version of Strimzi Kafka OAuth libraries.
+
+This image adds a `/opt/kafka/libs/oauth` directory, and copies the latest jars for OAuth support in it.
+Then it puts this directory as the first directory on the classpath.
+
+The result is that the most recent Strimzi Kafka OAuth jars and their dependencies are used, because they appear on the classpath before the ones that are part of `strimzi/kafka:latest-kafka-2.4.0` which are located in the `/opt/kafka/libs` directory.
+
+
+Building
+--------
+
+Use `docker build` to build the image:
+
+    docker build -t strimzi/kafka:latest-kafka-2.4.0-oauth .
+
+You can choose a different tag if you want.
+
+Also, take a look at Dockerfile:
+
+    less Dockerfile
+
+Note the `FROM` directive in the first line. It uses image coordinates to the latest publicly available Strimzi Kafka 2.4.0 image.
+
+You may want to adjust this to a different public image, or to one manually built previously and is only available in your private Docker Registry.
+
+For example, if you want to base your image on Strimzi Kafka 2.3.1 use `FROM strimzi/kafka:latest-kafka-2.3.1`.
+
+
+Validating
+----------
+
+You can start an interactive shell container and confirm that the jars are there.
+
+    docker run --rm -ti strimzi/kafka:latest-kafka-2.4.0-oauth /bin/sh
+    ls -la libs/oauth/
+    echo "$CLASSPATH"
+
+If you want to play around more within the container you may need to make yourself `root`.
+
+You achieve that by running the docker session as `root` user:
+
+    docker run --rm -ti --user root strimzi/kafka:latest-kafka-2.4.0-oauth /bin/sh
+
+
+
+Pushing the image to a Docker Repository
+--------------------------------------
+
+For Kubernetes to be able to use our image it needs to be pushed to either a public repository or to the private Docker Repository used by your Kubernetes distro.
+
+For example if you are using Kubernetes Kind as described in [HACKING.md](../../../HACKING.md) then your Docker Repository is listening on port 5000 of your local ethernet IP.
+
+    # On MacOS
+    export REGISTRY_IP=$(ifconfig en0 | grep 'inet ' | awk '{print $2}') && echo $REGISTRY_IP 
+
+    # On Linux
+    #export REGISTRY_IP=$(ifconfig docker0 | grep 'inet ' | awk '{print $2}') && echo $REGISTRY_IP 
+
+    export DOCKER_REG=$REGISTRY_IP:5000
+
+You need to retag the built image before so you can push it to Docker Registry:
+
+    docker tag strimzi/kafka:latest-kafka-2.4.0-oauth $DOCKER_REG/strimzi/kafka:latest-kafka-2.4.0-oauth
+    docker push $DOCKER_REG/strimzi/kafka:latest-kafka-2.4.0-oauth
+
+Actually, Kubernetes Kind supports an even simpler option how to make an image available to Kubernetes:
+
+    kind load docker-image strimzi/kafka:latest-kafka-2.4.0-oauth 
+
+Deploying
+---------
+
+In order for the operator to use your Kafka image, you have to replace the Kafka image coordinates in `install/cluster-operator/050-Deployment-strimzi-cluster-operator.yaml` in your `strimzi-kafka-operator` project.
+
+This image is based on `strimzi/kafka:latest-kafka-2.4.0`, so we need to replace all occurrences of that with the proper coordinates to our image:
+
+    sed -Ei 's#strimzi/kafka:latest-kafka-2.4.0#strimzi/kafka:latest-kafka-2.4.0-oauth#g' install/cluster-operator/050-Deployment-strimzi-cluster-operator.yaml
+
+You also have to push the image to the Docker Registry trusted by your Kubernetes cluster, and you need to adjust `050-Deployment-strimzi-cluster-operator.yaml` for changed coordinates due to that.
+
+For example:
+```
+sed -Ei -e "s#(image|value): strimzi/([a-z0-9-]+):latest#\1: ${DOCKER_REG}/strimzi/\2:latest#" \
+  -e "s#([0-9.]+)=strimzi/([a-zA-Z0-9-]+:[a-zA-Z0-9.-]+-kafka-[0-9.]+)#\1=${DOCKER_REG}/strimzi/\2#" \
+  install/cluster-operator/050-Deployment-strimzi-cluster-operator.yaml
+```
+
+It's best to check the `050-Deployment-strimzi-cluster-operator.yaml` file manually to make sure everything is in order:
+
+    less install/cluster-operator/050-Deployment-strimzi-cluster-operator.yaml
+
+
+You can now deploy Strimzi Kafka Operator following instructions in [HACKING.md](../../../HACKING.md)
+

examples/docker/strimzi-kafka-image/pom.xml

@@ -0,0 +1,71 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+    <modelVersion>4.0.0</modelVersion>
+
+    <parent>
+        <groupId>io.strimzi.oauth.docker</groupId>
+        <artifactId>kafka-oauth-docker-parent</artifactId>
+        <relativePath>../../pom.xml</relativePath>
+        <version>1.0.0-SNAPSHOT</version>
+    </parent>
+
+    <groupId>org.example</groupId>
+    <artifactId>kafka-oauth-docker-strimzi-kafka</artifactId>
+    <version>1.0.0-SNAPSHOT</version>
+
+    <packaging>pom</packaging>
+
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-dependency-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>copy</id>
+                        <phase>package</phase>
+                        <goals>
+                            <goal>copy</goal>
+                        </goals>
+                    </execution>
+                </executions>
+                <configuration>
+                    <artifactItems>
+                        <artifactItem>
+                            <groupId>io.strimzi</groupId>
+                            <artifactId>kafka-oauth-client</artifactId>
+                        </artifactItem>
+                        <artifactItem>
+                            <groupId>io.strimzi</groupId>
+                            <artifactId>kafka-oauth-server</artifactId>
+                        </artifactItem>
+                        <artifactItem>
+                            <groupId>io.strimzi</groupId>
+                            <artifactId>kafka-oauth-common</artifactId>
+                        </artifactItem>
+                        <artifactItem>
+                            <groupId>io.strimzi</groupId>
+                            <artifactId>kafka-oauth-keycloak-authorizer</artifactId>
+                        </artifactItem>
+                        <artifactItem>
+                            <groupId>org.keycloak</groupId>
+                            <artifactId>keycloak-core</artifactId>
+                        </artifactItem>
+                        <artifactItem>
+                            <groupId>org.keycloak</groupId>
+                            <artifactId>keycloak-common</artifactId>
+                        </artifactItem>
+                        <artifactItem>
+                            <groupId>org.bouncycastle</groupId>
+                            <artifactId>bcprov-jdk15on</artifactId>
+                        </artifactItem>
+                    </artifactItems>
+                    <outputDirectory>target/libs</outputDirectory>
+                    <overWriteReleases>false</overWriteReleases>
+                    <overWriteSnapshots>true</overWriteSnapshots>
+                </configuration>
+            </plugin>
+        </plugins>
+    </build>
+</project>

examples/kubernetes/README.md

@@ -0,0 +1,121 @@
+Examples of Strimzi Kafka Cluster with OAuth
+--------------------------------------------
+
+Here are several examples of Kafka Cluster definitions for deployment with Strimzi Cluster Operator.
+They assume Keycloak is used as an authorization server, with properly configured realms called 'demo', and 'authz'.
+
+* `keycloak.yaml`
+
+  A Keycloak pod you can use to start an ephemeral instance of Keycloak. Any changes to realms will be lost when the pod shuts down. This is the first yaml you'll want to deploy.
+
+* `kafka-oauth-singe.yaml`
+
+  A single node Kafka cluster using Apache Kafka 2.3.1 with OAuth 2 authentication using the 'demo' realm, and fast local signature validation (with keys loaded from the JWKS endpoint) for validating access tokens.
+
+* `kafka-oauth-single-2_4.yaml`
+
+  Same as `kafka-oauth-single.yaml` except using Apache Kafka 2.4.0.
+
+* `kafka-oauth-single-introspect.yaml`
+
+  A single node Kafka cluster using Apache Kafka 2.3.1 with OAuth 2 authentication using the `demo` realm, and introspection endpoint for access token validation.
+
+* `kafka-oauth-single-authz.yaml`
+
+  A single node Kafka cluster using Apache Kafka 2.3.1 with OAuth 2 authentication using the `kafka-authz` realm, a fast local signature validation, and Keycloak Authorization Services for token-based authorization.
+
+* `kafka-oauth-single-2_4.authz.yaml`
+
+  Same as `kafka-oauth-single-authz.yaml` except using Apache Kafka 2.4.0.
+
+### Deploying Keycloak and accessing the Keycloak Admin Console
+
+Before deploying any of the Kafka cluster definitions, you need to deploy a Keycloak instance, and configure the realms with the necessary client definitions.
+
+Deploy the Keycloak server:
+
+    kubectl apply -f keycloak.yaml 
+
+Wait for Keycloak to start up:
+
+    kubectl get pod
+    kubectl logs $(kubectl get pod | grep keycloak | awk '{print $1}')
+
+In order to connect to Keycloak Admin Console you need an ip address and a port where it is listening. From the point of view of the Keycloak pod it is listening on port 8080 on all the interfaces. The `NodePort` service also exposes a port on the Kubernetes Node's IP:
+
+    kubectl get svc | grep keycloak
+    KEYCLOAK_PORT=$(kubectl get svc | grep keycloak | awk -F '8080:' '{print $2}' | awk -F '/' '{print $1}')
+    echo Keycloak port: $KEYCLOAK_PORT 
+
+The actual IP address and port to use in order to reach Keycloak Admin Console from your host machine depends on your Kubernetes installation.
+
+
+#### Minishift
+
+    KEYCLOAK_HOST=$(minishift ip)
+    KEYCLOAK_PORT=$(kubectl get svc | grep keycloak | awk -F '8080:' '{print $2}' | awk -F '/' '{print $1}')
+    echo http://$KEYCLOAK_HOST:$KEYCLOAK_PORT/auth/admin
+
+You can then open the printed URL and login with admin:admin.
+
+
+#### Minikube
+
+You can connect directly to Kubernetes Node IP using a NodePort port:
+
+    KEYCLOAK_HOST=$(minikube ip)
+    KEYCLOAK_PORT=$(kubectl get svc | grep keycloak | awk -F '8080:' '{print $2}' | awk -F '/' '{print $1}')
+    echo http://$KEYCLOAK_HOST:$KEYCLOAK_PORT/auth/admin
+
+You can then open the printed URL and login with admin:admin.
+
+
+#### Kubernetes Kind
+
+In order to connect to Keycloak Admin Console you have to create a TCP tunnel:
+
+    kubectl port-forward svc/keycloak 8080:8080
+
+You can then open: http://localhost:8080/auth/admin and login with admin:admin.    
+
+
+### Importing example realms
+
+This step depends on your development environment because we have to build a custom docker image, and deploy it as a Kubernetes pod, for which we have to push it to the Docker Registry first.
+
+First we build the `keycloak-import` docker image:
+
+    cd examples/docker/keycloak-import
+    docker build . -t strimzi/keycloak-import
+
+Then we tag and push it to the Docker Registry:
+
+    docker tag strimzi/keycloak-import $REGISTRY_IP:$REGISTRY_PORT/strimzi/keycloak-import
+    docker push $REGISTRY_IP:$REGISTRY_PORT/strimzi/keycloak-import
+
+Here we assume we know the IP address (`$REGISTRY_IP`) of the docker container and the port (`$REGISTRY_PORT`) it's listening on, and that, if it is an insecure Docker Registry, the Docker Daemon has been configured to trust the insecure registry. We also assume that you have authenticated to the registry if that is required in your environment. And, very important, we also assume that this is either a public Docker Registry accessible to your Kubernetes deployment or that it's the internal Docker Registry used by your Kubernetes install.
+
+See [HACKING.md](../../HACKING.md) for more information on setting up the local development environment with all the pieces in place.
+
+
+Now deploy it as a Kubernetes pod:
+
+    kubectl run -ti --attach keycloak-import --image=$REGISTRY_IP:$REGISTRY_PORT/strimzi/keycloak-import
+
+The continer will perform the imports of realms into the Keycloak server, and exit. If you run `kubectl get pod` you'll see it CrashLoopBackOff because as soon as it's done, Kubernetes will restart the pod in the background, which will try to execute the same imports again, and fail. You'll also see errors in the Keycloak log, but as long as the initial realm import was successful, you can safely ignore them.
+
+Remove the `keycloak-import` deployment:
+
+    kubectl delete deployment keycloak-import
+
+
+### Deploying the Kafka cluster
+
+Assuming you have already installed Strimzi Kafka Operator, you can now simply deploy one of the `kafka-oatuh-*` yaml files. All examples are configured with OAuth2 for authentication.
+
+For example:
+
+    kubectl apply -f kafka-oauth-single-authz.yaml
+
+
+