This folder contains example applications connecting to StreamNative Cloud to demonstrate the use of Sarama. For code snippet examples on how to use the different types in Sarama, see Sarama's API documentation on pkg.go.dev
-
Get the bootstrap servers. Note down the bootstrap servers as you'll need them in the next steps.
-
Create a service account and get the API key. For simplicity, you can create a super-user service account to run all the examples without worrying about authorization settings.
-
Clone the repo.
git clone https://github.com/streamnative/examples.git
-
Enter the
examples/cloud_kafka/sarama
directory.cd examples/cloud_kafka/sarama
-
Build the examples.
sh build.sh
In the terminal you are running the examples from, export the following environment variables with the values for your StreamNative Cloud cluster.
export BOOTSTRAP_SERVERS="<your-bootstrap-servers>"
export API_KEY="<your-api-key>"
The following examples are adapted from the Sarama Examples
consumer_group is a simple example that demonstrates how to use the Sarama consumer group client to consume messages from a Kafka topic.
./consumergroup/consumergroup -brokers=$BOOTSTRAP_SERVERS -apiKey=$API_KEY -topics="sarama-test" -group="sarama-example"
http_server is a simple HTTP server uses both the sync producer to produce data as part of the request handling cycle, as well as the async producer to maintain an access log. It also uses the mocks subpackage to test both.
./http_server/http_server -brokers=$BOOTSTRAP_SERVERS -apiKey=$API_KEY
Basic example to use a producer interceptor that produces OpenTelemetry spans and add some headers for each intercepted message.
./interceptors/interceptors -brokers=$BOOTSTRAP_SERVERS -apiKey=$API_KEY -topic="sarama-interceptor"
txn_producer Basic example to use a transactional producer that produce on some topic within a Kafka transaction. To ensure transactional-id uniqueness it implement some ProducerProvider that build a producer appending an integer that grow when producer is created.
./txn_producer/txn_producer -brokers $BOOTSTRAP_SERVERS -apiKey $API_KEY -producers 3 -records-number 10 -topic txn_topic
exactly_once Basic example to use a transactional producer that produce consumed message from some topics within a Kafka transaction. To ensure transactional-id uniqueness it implement some ProducerProvider that build a producer using current message topic-partition.
./exactly_once/exactly_once -brokers $BOOTSTRAP_SERVERS -apiKey $API_KEY -topics eo_input_topic -destination-topic eo_output_topic -group eo_sub -verbose
The above command reads messages from eo_input_topic
and write messages to eo_output_topic
in a transactional way.
Once this application is running, you can use txn_producer
to produce messages to the eo_input_topic
.
./txn_producer/txn_producer -brokers $BOOTSTRAP_SERVERS -apiKey $API_KEY -producers 3 -records-number 1ic eo_input_topic
Then the messages will be claimed by exactly_once
and produced to the output eo_output_topic
.
Then you can use consumergroup
example to read the messages from eo_output_topic
.
./consumergroup/consumergroup -brokers $BOOTSTRAP_SERVERS -apiKey $API_KEY -group reader -topics eo_output_topic -verbose
Then you should be able to see similar output below in the terminal where you run consumergroup
example:
2024/11/17 14:15:35 Message claimed: value = test, timestamp = 2024-11-17 14:12:52.228 -0800 PST, topic = eo_output_topic
2024/11/17 14:15:35 Message claimed: value = test, timestamp = 2024-11-17 14:13:03.432 -0800 PST, topic = eo_output_topic
2024/11/17 14:15:35 Message claimed: value = test, timestamp = 2024-11-17 14:13:09.036 -0800 PST, topic = eo_output_topic
2024/11/17 14:15:35 Message claimed: value = test, timestamp = 2024-11-17 14:13:15.174 -0800 PST, topic = eo_output_topic
2024/11/17 14:15:35 Message claimed: value = test, timestamp = 2024-11-17 14:13:21.746 -0800 PST, topic = eo_output_topic
2024/11/17 14:15:35 Message claimed: value = test, timestamp = 2024-11-17 14:13:28.075 -0800 PST, topic = eo_output_topic
2024/11/17 14:15:35 Message claimed: value = test, timestamp = 2024-11-17 14:13:33.647 -0800 PST, topic = eo_output_topic
2024/11/17 14:15:35 Message claimed: value = test, timestamp = 2024-11-17 14:13:39.305 -0800 PST, topic = eo_output_topic
2024/11/17 14:15:35 Message claimed: value = test, timestamp = 2024-11-17 14:13:45.365 -0800 PST, topic = eo_output_topic
produce is a simple example that demonstrates how to use the Sarama sync/async/batch producers to send messages to a Kafka topic
./produce/produce -brokers=$BOOTSTRAP_SERVERS -apiKey=$API_KEY -topics="sarama-produce"