How Copilot Simplifies the Painful Process of Software Installation: A Case Study with Apache Kafka

The Challenge of Installing Software

Installing software is often a pain point for many developers. This is because software is constantly evolving, with new versions being released regularly. As a result, tutorials and guides can quickly become outdated. This is especially true for complex software like Apache Kafka, which has many dependencies and configuration steps.

Moreover, we can’t blindly follow Language Learning Models (LLMs) because they are also outdated most of the time. Therefore, we need to be sure to ask the correct questions and cross-verify the information with the latest official documentation. I have tried for this process 3 LLMs but I was the most satisfied with the results from Copilot. 

In this blog post, we are going to install Apache Kafka as part of a project. The goal is to provide an updated guide based on the latest version of Kafka and Ubuntu LTS. Let’s get started! We will let copilot start with describing its strategy and next we also provide the updated guide. Also for now this is still a static guide but for the future we will explore in making this dynamic by using the latest technologies.

The thought process of Copilot

I make the follwing text in Italic because this was the way that Copilot decided it want to answer the question.

As an AI developed by OpenAI, I’m here to share how I can assist with a wide range of tasks, including providing updated information based on the context provided. Today, we’re going to tackle a common pain point for many developers: installing the latest version of software.

In this case, we’re focusing on Apache Kafka, a popular open-source distributed event streaming platform. A user provided me with the content of two tutorials and asked for an updated version. Rather than copying or directly modifying the tutorials, I applied my understanding of the task to generate a new, relevant set of instructions.

Here’s how I approached it:

1. Understanding the Task: The first step was to understand the user’s request. They wanted to install Apache Kafka on Ubuntu 22.04.4 LTS with JDK 22, but the tutorials they had were outdated.

2. Applying Internal Knowledge: I used my internal knowledge, which includes general procedures for installing software like Apache Kafka, to create an updated guide. This didn’t involve copying or directly modifying the tutorials provided, but rather applying my understanding of the task to generate a new, relevant set of instructions.

3. Providing a Guide, Not a Definitive Source: While I strive to provide accurate and helpful information, my responses should be used as a guide and not a definitive source. Always refer to official documentation and resources when available, especially for complex tasks like software installation.

By following these steps, I was able to provide the user with an updated guide for installing Apache Kafka. This is just one example of how AI can simplify complex tasks and reduce pain points for developers.

The tutorial

Absolutely, I’d be happy to provide more context for each step. Here’s the updated guide with additional explanations:

1. Update the system packages This is a standard step in preparing for any software installation on Ubuntu. It ensures that all existing packages on the system are up-to-date.
   
   sudo apt update

2. Install JDK 21 Apache Kafka requires Java to run. The version of Java you need can vary, but here we’re using Java 22 as per your request.
   
   sudo apt install openjdk-21jdk

3. Download and unzip the Apache Kafka binary file This step involves downloading the latest version of Apache Kafka from the official website and unzipping it. We then move the unzipped directory to /opt/kafka, which is a common location for optional software on Unix-like systems.
   
   wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz 

     tar xzf kafka_2.13-3.7.0.tgz
    sudo mkdir /opt/kafka 

    sudo mv kafka_2.13-3.7.0  /opt/kafka

4. Create and reload the systemd unit files for Zookeeper and Kafka services Systemd is a system and service manager for Linux. It allows you to control and manage services (like Kafka and Zookeeper) as system services. Here, we’re creating systemd unit files for both Zookeeper and Kafka, and then reloading systemd to recognize these new services.
   
   sudo nano /etc/systemd/system/zookeeper.service
   and include the following content
   
   [Unit]

Description=Apache Zookeeper server

Documentation=http://zookeeper.apache.org

Requires=network.target remote-fs.target

After=network.target remote-fs.target

[Service]

Type=simple

ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties

ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh

Restart=on-abnormal

[Install]

WantedBy=multi-user.target

sudo nano /etc/systemd/system/kafka.service

[Unit]

Description=Apache Kafka Server

Documentation=http://kafka.apache.org/documentation.html

Requires=zookeeper.service

[Service]

Type=simple

ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

ExecStop=/opt/kafka/bin/kafka-server-stop.sh

Restart=on-abnormal

[Install]

WantedBy=multi-user.target

sudo systemctl daemon-reload

5. Start Zookeeper and Kafka services and check their status Now that systemd knows about Zookeeper and Kafka, we can start these services and check their status to ensure they’re running properly.
   
   sudo systemctl start zookeeper

sudo systemctl start kafka

6. Create a topic in Kafka Topics in Kafka are categories or feeds to which messages are published. Here, we’re creating a topic named ‘test’.
   
   /opt/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

7. Send and read messages Finally, we’re sending a message to the ‘test’ topic using the Kafka console producer script, and then reading it back using the Kafka console consumer script.
   
     echo "Hello, World" | /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test > /dev/null

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

This was a last remark from Copilot

Remember, these steps are a general guide and might need to be adjusted based on your specific setup and requirements. Always refer to the official documentation for the most accurate and up-to-date information. If you encounter any issues, feel free to ask for further assistance. Happy coding! 😊