Kafka Deployment Task

This tutorial follows the Zookeeper deployment and guides you through deploying Apache Kafka, a distributed event streaming platform. It is designed for students and junior DevOps engineers learning to deploy stateful, networked applications on Kubernetes with production-like configurations.

By completing this task, you will create a Kubernetes Deployment for Kafka with specific performance and configuration settings, and expose it for internal cluster communication with a ClusterIP Service.

In this tutorial, you will be tasked to:

Translate a detailed list of Kafka configuration needs into a Kubernetes manifest.
Configure environment variables for connectivity, listener protocols, and performance tuning (e.g., JVM heap size and log retention).
Create a ClusterIP service to enable communication within the Kubernetes cluster.
Prepare and apply the manifests needed to deploy Kafka.

Before you start

Make sure that:

The Zookeeper Deployment and Service from the previous task are running successfully in the cluster.
You are SSHed into the k3smain node and have switched to the shared user account.

Part 1: Create the Kafka Deployment Manifest

Your first goal is to create a Kubernetes deployment file named kafka-deployment.yaml. This deployment will run the Kafka broker pod with a specific set of configurations.

Deployment Requirements

The Kafka instance must be configured with the following specifications:

Replica Count: The deployment should manage a single (1) replica.
Node Affinity: The pod must be scheduled to run on the node named k3smain.
Container Image: Use the specific bitnami/kafka:3.9 image.
Resource Management: The container must be allocated specific CPU and memory resources:
- Requests: 1.5Gi of memory and 1 CPU.
- Limits: 2Gi of memory and 1 CPU.
Storage: For this simulation, Kafka should use a non-persistent, ephemeral volume (emptyDir) mounted at /bitnami/kafka.
Port: The container must expose the standard Kafka port, 9092.
Environment Configuration: The following environment variables must be set to configure the Kafka broker:
- KAFKA_ZOOKEEPER_CONNECT_TIMEOUT_MS: Set to 20000.
- KAFKA_CFG_ZOOKEEPER_CONNECT: Set to zookeeper-service:2181 to connect to the Zookeeper service.
- ALLOW_PLAINTEXT_LISTENER: Must be set to yes.
- KAFKA_CFG_LISTENERS: Configure the internal listener to PLAINTEXT://:9092.
- KAFKA_CFG_ADVERTISED_LISTENERS: Advertise the in-cluster address as PLAINTEXT://kafka-service:9092.
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: Map the listener to its protocol with PLAINTEXT:PLAINTEXT.
- KAFKA_HEAP_OPTS: Set the Java heap size with -Xms1536m -Xmx1536m.
- KAFKA_CFG_LOG_RETENTION_MS: Set to 300000.
- KAFKA_CFG_LOG_SEGMENT_MS: Set to 300000.
- KAFKA_CFG_LOG_SEGMENT_BYTES: Set to 104857600.

Skeleton File for Deployment

Create a file named kafka-deployment.yaml and fill it out according to the detailed requirements above.

# kafka-deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka
  labels:
    app: kafka
spec:
  # Requirement 1: Add replica count
  replicas: <Your-Replica-Count>
  selector:
    matchLabels:
      app: kafka
  template:
    metadata:
      labels:
        app: kafka
    spec:
      # Requirement 2: Add node selector
      nodeSelector:
        kubernetes.io/hostname: <Your-Node-Name>
      containers:
      - name: kafka
        # Requirement 3: Specify exact container image and tag
        image: <Your-Image-Name>:<Your-Tag>
        imagePullPolicy: Always
        ports:
        # Requirement 6: Expose the container port
        - containerPort: <Your-Port>
        # Requirement 7: Add all 10 required environment variables
        env:
        - name: KAFKA_ZOOKEEPER_CONNECT_TIMEOUT_MS
          value: <Value>
        - name: KAFKA_CFG_ZOOKEEPER_CONNECT
          value: <Value>
        # ... add all other env vars here ...
        volumeMounts:
        # Requirement 5: Mount the data volume
        - name: data
          mountPath: <Your-Mount-Path>
        # Requirement 4: Add resource requests and limits
        resources:
          requests:
            memory: <Memory-Request>
            cpu: <CPU-Request>
          limits:
            memory: <Memory-Limit>
            cpu: <CPU-Limit>
      volumes:
      # Requirement 5: Define the ephemeral volume
      - name: data
        emptyDir: {}

Part 2: Expose Kafka with a Service

Because Kafka's ADVERTISED_LISTENERS setting points to kafka-service, you need to create a service with that exact name so other pods in the cluster can resolve it. A ClusterIP service is appropriate for this internal-only communication.

Service Requirements

The Service must:

Have the name kafka-service.
Be of type ClusterIP, which is the default service type.
Expose port 9092 and forward traffic to targetPort 9092.
Use a selector to correctly identify the Kafka pod (app: kafka).

Skeleton File for Service

Create a second file named kafka-service.yaml and complete the skeleton below.

# kafka-service.yaml
---
apiVersion: v1
kind: Service
metadata:
  # Requirement 1: Specify the service name
  name: <Your-Service-Name>
  labels:
    app: kafka
spec:
  # Requirement 2: Specify the service type (or leave default for ClusterIP)
  type: <Your-Service-Type>
  ports:
  # Requirement 3: Define port and targetPort
  - port: <Service-Port>
    targetPort: <Pod-Port>
  # Requirement 4: Add the correct selector
  selector:
    app: <Your-App-Label>

Part 3: Deployment and Verification

Once you have created and completed both YAML files, apply them and check the logs to ensure the Kafka broker starts successfully and connects to Zookeeper.

Apply the manifests: kubectl apply -f kafka-deployment.yaml kubectl apply -f kafka-service.yaml
Verify the deployment: kubectl get pods -l app=kafka
Check for errors: kubectl logs <pod name>

Once the deployment is successful, you can proceed to the Streamer task.

Solutions

# kafka-deployment.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: kafka labels: app: kafka spec: replicas: 1 selector: matchLabels: app: kafka template: metadata: labels: app: kafka spec: nodeSelector: kubernetes.io/hostname: k3smain containers: - name: kafka image: bitnami/kafka:3.9 imagePullPolicy: Always ports: - containerPort: 9092 env: - name: KAFKA_ZOOKEEPER_CONNECT_TIMEOUT_MS value: "20000" - name: KAFKA_CFG_ZOOKEEPER_CONNECT value: "zookeeper-service:2181" - name: ALLOW_PLAINTEXT_LISTENER value: "yes" - name: KAFKA_CFG_LISTENERS value: "PLAINTEXT://:9092" - name: KAFKA_CFG_ADVERTISED_LISTENERS value: "PLAINTEXT://kafka-service:9092" - name: KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP value: "PLAINTEXT:PLAINTEXT" - name: KAFKA_HEAP_OPTS value: "-Xms1536m -Xmx1536m" - name: KAFKA_CFG_LOG_RETENTION_MS value: "300000" - name: KAFKA_CFG_LOG_SEGMENT_MS value: "300000" - name: KAFKA_CFG_LOG_SEGMENT_BYTES value: "104857600" volumeMounts: - name: data mountPath: /bitnami/kafka resources: requests: memory: "1.5Gi" cpu: "1" limits: memory: "2Gi" cpu: "1" volumes: - name: data emptyDir: {}

# kafka-service.yaml --- apiVersion: v1 kind: Service metadata: name: kafka-service labels: app: kafka spec: type: ClusterIP ports: - port: 9092 targetPort: 9092 selector: app: kafka

Last modified: 22 June 2025