Basic setup of a Multi Node Apache Kafka/Zookeeper Cluster


Install three nodes with CentOS 7 with at least 20GB Disk, 2 GB RAM and two CPU Cores.

Install JDK

Set JAVA_HOME in ~/.bashrc

Disable SELinux, Firewall and IPv6

Reboot Server

Installing Kafka

Download Kafka and unpack it under /opt

Starting Zookeeper

On each node create a zookeeper directory and a file ‚myid‘ with a unique number:

On all three Server go to Kafka home folder /opt/kafka_2.11- and setup zookeeper like this

vi config/

Start Zookeeper on all three servers

Change the Kafka on all three servers (set a unique broker id on each server)

vi config/

Start Kafka on all three nodes:

Verify kafka and zookeper are running:

Verify also all brokers are registered to zookeeper:

Create a example Topic with three partitions and replicationfactor 3

Test the Topic

Start a Producer on one node:

Start also a Consumer on a different node:

Write some text in the producer console. You should then see the Text on the Consumer Console.

Stop a node and write again some messages in the producer console to verify the high availability is working.