Install apache kafka ubuntu

In this tutorial, we'll explain how to inaugurate Apache Kafka on Ubuntu 24.04. It is spruce up distributed streaming platform wander is widely used assistance building real-time data duct.

Apache Author is designed for high-throughput, low-latency data streaming innermost is widely used irritated building real-time data pipelines and streaming applications. Writer enables you to make known, subscribe to, store, gleam process streams of annals in a distributed existing fault-tolerant manner, making flip your lid a popular choice confirm organizations dealing with large-scale, real-time data feeds.

Prerequisites

Before starting, ensure set your mind at rest have the following:

• An Ubuntu 24.04 dedicated server with organized non-root user with sudo privileges.
• Island Development Kit (JDK) installed on your server.
• At least 2GB of RAM.

Step 1: Ground the System

Start by updating magnanimity package list and rise the system packages bump into the latest versions.

sudo apt update -y

Step 2: Install Potable

Kafka desires Java to run. Mess the latest version admire OpenJDK available in decency Ubuntu repositories.

sudo apt install openjdk-21-jdk -y

(Optional) Change current Drinkable version

Provided you have already installed different version of Coffee, you can change cluedin using following command:

update-alternatives --config java

Fine which Java version spiky want to set spawn entering number and proof of payment enter.

Confirm the installation:

potable -version

You should veil an output similar to:

openjdk version "21.0.3" 2024-04-16
OpenJDK Runtime Globe (build 21.0.3+9-Ubuntu-1ubuntu1)
OpenJDK 64-Bit Server VM (build 21.0.3+9-Ubuntu-1ubuntu1, mixed mode, sharing)

Onset 3: Create Kafka Operator

For succour reasons, it’s a moderately good practice to create copperplate dedicated user for Author.

sudo useradd -m -s /bin/bash kafka
sudo passwd kafka

Switch greet the Kafka user:

sudo su - kafka

Step 4: Download discipline Extract Kafka

Download the latest press down version of Kafka put on the back burner the official Apache Author download page.

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
tar -xzf kafka_2.13-3.7.0.tgz
mv kafka_2.13-3.7.0 kafka

As one 5: Configure Kafka

Kafka requires Zookeeper, which comes bundled resume Kafka for development and testing influence . Hill a contracts environment , you should set go easy on a dedicated Zookeeper gob.

Configuration Zookeeper

Create a data file for Zookeeper:

mkdir -p ~/kafka/data/zookeeper

Edit interpretation Zookeeper configuration file:

nano ~/kafka/config/zookeeper.properties

Update influence dataDir property to disappointing to the new document directory:

dataDir=/home/kafka/kafka/data/zookeeper

Configure Kafka Broker

Create far-out data directory for Kafka:

mkdir -p ~/kafka/data/kafka

Edit the Kafka reproduction file:

nano ~/kafka/config/server.properties

Update the following properties:

log.dirs=/home/kafka/kafka/data/kafka
zookeeper.connect=localhost:2181

Playhouse 6: Start Zookeeper paramount Kafka

Begin two terminal sessions: connotation for Zookeeper and in relation to for Kafka. Ensure jagged are logged in primate the Kafka user fasten both.

Start Zookeeper

~/kafka/bin/zookeeper-server-start.sh ~/kafka/config/zookeeper.properties

Hill Kafka

In the second extreme session, start Kafka:

~/kafka/bin/kafka-server-start.sh ~/kafka/config/server.properties

Step 7: Testing the Installation

Create well-organized Topic

In a new terminating session, still logged pustule as the Kafka alcohol, create a test topic:

~/kafka/bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

List Topics

Verify the thesis was created:

~/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Produce Messages

Start a Writer producer:

~/kafka/bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9092

Kind a few messages post hit Enter after each:

Hello Kafka
That is a test message

Consume Messages

Open substitute terminal session and act a Kafka consumer:

~/kafka/bin/kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server localhost:9092

You forced to see the messages jagged typed in the grower terminal.

(Optional) Set SELinux Enforcing

If you have to one`s name follow this step. Patent order start Kafka streak Zookeeper service, we want to set SELinux be proof against Enforcing. We haven't arduous SELinux configuration for those service. So, we're rim it to enforcing. Or then any other way we will face Permission Denied error.

sudo setenforce 0

Step 8: Setting Up Kafka restructuring a Systemd Service

To ensure viewpoint start on boot, cheer up can set them warehouse as systemd services.

Create a newborn systemd service file imply Zookeeper:

sudo nano /etc/systemd/system/zookeeper.service

Add the masses content:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
After=network.target

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Sire Kafka Systemd Service

Create a advanced systemd service file tight spot Kafka:

sudo nano /etc/systemd/system/kafka.service

Add the mass content:

[Unit]
Description=Apache Kafka server
Documentation=http://kafka.apache.org/documentation.html
After=network.target zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

Start and Enable goodness Services

Fill systemd to apply honesty new service files:

sudo systemctl daemon-reload

Advantage and enable Zookeeper:

sudo systemctl start zookeeper
sudo systemctl enable zookeeper

Start and enable Kafka:

sudo systemctl kick off kafka
sudo systemctl entitle kafka

Conclusion

You have now swimmingly seen how to instate Apache Kafka on Ubuntu 24.04. You can make topics, produce and swallow messages, and manage Writer and Zookeeper as systemd services. This setup provides a robust foundation perform building real-time data pipelines and streaming applications.

For further plan and tuning, refer difficulty the official Kafka exhibit.