kafka role

[ryane]

• Installs cleanly on a fresh build of most recent master branch
• Upgrades cleanly from the most recent release
• Updates documentation relevant to the changes

Installation documentation available in the role readme. Note that you will need at least 3 workers each with 1 CPU and 4 GB of memory available to Mesos in order to run with the default configuration.

This depends on mantl/nginx-mantlui#18 being merged and the 0.6.8 tag of ciscocloud/nginx-mantlui being pushed to docker hub.

After successful Kafka installation (it can take a few minutes), verify the following:

• that the mantl/kafka scheduler is running and healthy in Marathon
• that there are 3 Kafka brokers running in Mesos (broker-1, broker-2, broker-3)

Once the brokers are up, you should be able to produce and consume data. Here is one example of a manual test:

# create a topic
curl -sku admin:password -d "topic=journal" https://control-node-01/kafka/api/topic/add

# on one node, pipe journald logs into the journal kafka topic
sudo journalctl -f | sudo docker run --rm -i ryane/kafkacat -b broker-1.service.consul:9092,broker-2.service.consul:9092,broker-3.service.consul:9092 -t journal -z snappy

# on another node, consume the events from the  journal topic
sudo docker run --rm -it ryane/kafkacat -b broker-1.service.consul:9092,broker-2.service.consul:9092,broker-3.service.consul:9092 -t journal

[tpolekhin]

too many CPUs for only 4GB ram,
lets follow the 1/4 proportion (1cpu/4gb, 2cpu/8gb, ...)

[ryane]

thanks, I updated the config as suggested

[langston-barrett]

We can see if the tag has been pushed on the nginx-mantlui docker hub page.

[ryane]

0.6.8 tag of the nginx-mantlui image has been published

[langston-barrett]

I wonder if this could be broken up into a dict to be joined by templating, i.e.

kafka_broker_options:
  log.flush.interval.ms: "10000"
  num.recovery.threads.per.data.dir: "1"
  ...

[stevendborrelli]

I agree with this, it is hard to parse this command

[stevendborrelli]

We probably don't want to use zookeeper.service.consul, as the zk client does not refresh DNS entries, so we lose benefits of RR dns. The zk hosts should be enumerated.

[tpolekhin]

@stevendborrelli

the zk client does not refresh DNS entries

THAT explains a lot... :)

[ryane]

kafka will start using the list of zk hosts once #1308 is merged