Setting Up Kafka Locally (Docker & Native)

Before diving into Kafka development, you need a local Kafka instance to experiment and build with. In this post, we'll walk through two ways to set up Kafka on your machine: using Docker (the easiest way) and installing it natively.

Option 1: Kafka with Docker Compose

Recommended if: You want a quick, isolated setup without polluting your system.

Prerequisites

• Docker Desktop
• Docker Compose

Create a docker-compose.yml

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.6.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:7.6.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  

Start the cluster

docker-compose up -d

You now have Kafka running on localhost:9092.

Option 2: Install Kafka Natively

Recommended if: You want full control or to understand the internals.

Prerequisites

• Java 8 or later
• Curl or wget

Download and Extract Kafka

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
tar -xvzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0
  

Start Zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

Start Kafka Broker

bin/kafka-server-start.sh config/server.properties

Kafka will now be running at localhost:9092.

Testing Your Setup

Create a Topic

bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Produce a Message

bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092

Type a message like Hello Kafka and hit Enter.

Consume the Message

bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092

You should see the message appear in the console!

Tips

• Use docker-compose logs kafka to debug if Kafka fails to start.
• Kafka logs are verbose; check them for port binding or configuration issues.
• Kafka won’t work well if ports like 2181 or 9092 are blocked or already in use.

Conclusion

Now you have a working Kafka instance on your machine! You can use this environment to build producers, consumers, and explore Kafka’s core APIs. In the next post, we'll write our first Kafka Producer and Consumer using Java.