Installing Kafka

The current version of ECiDA uses kafka to communicate between module(-sidecar)s. To install kafka onto the cluster, it is easiest to use the Strimzi (opens new window) operator. During development we have not yet set any requirements of our Kafka cluster, other than that it is reachable.

Cluster requirements

Strimzi.io recommends at least 4GiB of memory for develoment Kubernetes clusters for the Kafka operator to run well. Make sure that when using minikube on a VM driver that it is started with enough memory, e.g. minikube start --memory 6000.

Installing the operator

To install the operator, run the following command:

kubectl create namespace kafka

kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

The Strimzi operator image is quite large, so the download may take a little while. Run kubectl get pods -n kafka --watch to see when the operator has started successfully. Once it has started, proceed to the next section to instantiate a kafka cluster.

Creating a cluster

Once the operator is created and is up and running, you can deploy an instance of the Kafka CRD. Run the following command to instantiate a Kafka cluster with 1 kafka replica and 1 zookeeper replica.

# Note: we have our own CR for creating clusters. You probably do not want to
# use this after the next documentation chapter.
kubectl apply -f https://strimzi.io/examples/latest/kafka/kafka-persistent-single.yaml -n kafka 

Creating a cluster 2

In the next chapter: Testing kafka related code locally we run tests against a kafka cluster locally. In this chapter we apply a modified version of kafka-persistent-single.yaml that exposes brokers on a loadbalancer. You probably want to use that one instead of the one from https://strimzi.io.

More information

This information was taken from the Strimzi quickstart (opens new window) webpage.