Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(confluent): add details for actions pod for confluent #6810

Merged
merged 4 commits into from
Dec 22, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions docs/deploy/confluent-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,44 @@ container.

If you're deploying with Docker Compose, you do not need to deploy the Zookeeper, Kafka Broker, or Schema Registry containers that ship by default.

#### DataHub Actions

Configuring Confluent Cloud for DataHub Actions requires some additional edits to your `executor.yaml`. Under the Kafka
source connection config you will need to add the Python style client connection information:

```yaml
connection:
bootstrap: ${KAFKA_BOOTSTRAP_SERVER:-localhost:9092}
schema_registry_url: ${SCHEMA_REGISTRY_URL:-http://localhost:8081}
consumer_config:
security.protocol: ${KAFKA_PROPERTIES_SECURITY_PROTOCOL:-PLAINTEXT}
sasl.mechanism: ${KAFKA_PROPERTIES_SASL_MECHANISM:-PLAIN}
sasl.username: ${KAFKA_PROPERTIES_SASL_USERNAME}
sasl.password: ${KAFKA_PROPERTIES_SASL_PASSWORD}
schema_registry_config:
basic.auth.user.info: ${KAFKA_PROPERTIES_BASIC_AUTH_USER_INFO}
```

Specifically `sasl.username` and `sasl.password` are the differences from the base `executor.yaml` example file.

Additionally, you will need to set up environment variables for `KAFKA_PROPERTIES_SASL_USERNAME` and `KAFKA_PROPERTIES_SASL_PASSWORD`
which will use the same username and API Key you generated for the JAAS config.

Next, configure datahub-frontend to connect to Confluent Cloud by changing `docker/datahub-actions/env/docker.env`:

```
KAFKA_BOOTSTRAP_SERVER=pkc-g4ml2.eu-west-2.aws.confluent.cloud:9092

# Confluent Cloud Configs
KAFKA_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL
KAFKA_PROPERTIES_SASL_USERNAME=XFA45EL1QFUQP4PA
KAFKA_PROPERTIES_SASL_PASSWORD=ltyf96EvR1YYutsjLB3ZYfrk+yfCXD8sQHCE3EMp57A2jNs4RR7J1bU9k6lM6rU
KAFKA_PROPERTIES_SASL_MECHANISM=PLAIN
KAFKA_PROPERTIES_CLIENT_DNS_LOOKUP=use_all_dns_ips
KAFKA_PROPERTIES_BASIC_AUTH_CREDENTIALS_SOURCE=USER_INFO
KAFKA_PROPERTIES_BASIC_AUTH_USER_INFO=P2ETAN5QR2LCWL14:RTjqw7AfETDl0RZo/7R0123LhPYs2TGjFKmvMWUFnlJ3uKubFbB1Sfs7aOjjNi1m23
```

### Helm

If you're deploying on K8s using Helm, you can simply change the **datahub-helm** `values.yml` to point to Confluent Cloud and disable some default containers:
Expand Down Expand Up @@ -143,6 +181,41 @@ springKafkaConfigurationOverrides:

Then simply apply the updated `values.yaml` to your K8s cluster via `kubectl apply`.

#### DataHub Actions

Configuring Confluent Cloud for DataHub Actions requires some additional edits to your `executor.yaml`. Under the Kafka
source connection config you will need to add the Python style client connection information:

```yaml
connection:
bootstrap: ${KAFKA_BOOTSTRAP_SERVER:-localhost:9092}
schema_registry_url: ${SCHEMA_REGISTRY_URL:-http://localhost:8081}
consumer_config:
security.protocol: ${KAFKA_PROPERTIES_SECURITY_PROTOCOL:-PLAINTEXT}
sasl.mechanism: ${KAFKA_PROPERTIES_SASL_MECHANISM:-PLAIN}
sasl.username: ${KAFKA_PROPERTIES_SASL_USERNAME}
sasl.password: ${KAFKA_PROPERTIES_SASL_PASSWORD}
schema_registry_config:
basic.auth.user.info: ${KAFKA_PROPERTIES_BASIC_AUTH_USER_INFO}
```

Specifically `sasl.username` and `sasl.password` are the differences from the base `executor.yaml` example file.

Additionally, you will need to set up secrets for `KAFKA_PROPERTIES_SASL_USERNAME` and `KAFKA_PROPERTIES_SASL_PASSWORD`
jjoyce0510 marked this conversation as resolved.
Show resolved Hide resolved
which will use the same username and API Key you generated for the JAAS config.

```yaml
credentialsAndCertsSecrets:
name: confluent-secrets
secureEnv:
sasl.jaas.config: sasl_jaas_config
basic.auth.user.info: basic_auth_user_info
sasl.username: sasl_username
sasl.password: sasl_password
```

The Actions pod will automatically pick these up in the correctly named environment variables when they are named this exact way.

## Contribution
Accepting contributions for a setup script compatible with Confluent Cloud!

Expand Down