Basic setup of a Multi Node Apache Kafka/Zookeeper Cluster

Prerequesites

Install three nodes with CentOS 7 with at least 20GB Disk, 2 GB RAM and two CPU Cores.

Install JDK

Set JAVA_HOME in ~/.bashrc

Disable SELinux, Firewall and IPv6

Reboot Server

Installing Kafka

Download Kafka and unpack it under /opt

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.2/kafka_2.11-0.11.0.2.tgz

Starting Zookeeper

On each node create a zookeeper directory and a file ‚myid‘ with a unique number:

On all three Server go to Kafka home folder /opt/kafka_2.11-0.11.0.1 and setup zookeeper like this

vi config/zookeeper.properties

Start Zookeeper on all three servers

Change the Kafka server.properties on all three servers (set a unique broker id on each server)

vi config/server.properties

Start Kafka on all three nodes:

Verify kafka and zookeper are running:

Verify also all brokers are registered to zookeeper:

Create a example Topic with three partitions and replicationfactor 3

Test the Topic

Start a Producer on one node:

Start also a Consumer on a different node:

Write some text in the producer console. You should then see the Text on the Consumer Console.

Stop a node and write again some messages in the producer console to verify the high availability is working.