java code with proto and Gradle

 

Kafka Protobuf Consumer

This project demonstrates how to consume messages from a Kafka topic using a Java application. The messages are serialized using Protocol Buffers (Protobuf) and deserialized in the application.

Prerequisites

1. Java Development Kit (JDK): Install JDK 11 or later.
2. Apache Kafka: Ensure you have access to a Kafka cluster.
3. Protocol Buffers Compiler (protoc):
• Download the correct version for your platform from the Protocol Buffers GitHub releases.
• Add the protoc binary to your system PATH.
• Verify installation:

bash

 

protoc --version

4. Maven or Gradle: Install Maven or Gradle to manage project dependencies.

---

Setup Instructions

Step 1: Clone or Download the Project

bash

 

git clone <repository-url>

cd KafkaProtobufConsumer

---

Step 2: Compile Protobuf File

1. Place your .proto file (e.g., PPP_Cloud_Streaming.proto) in the src/main/proto directory.
2. Use protoc to generate Java files:

bash

 

protoc --java_out=src/main/java src/main/proto/PPP_Cloud_Streaming.proto

3. The generated Java files will be placed in the src/main/java directory under the specified package.

---

Step 3: Configure Project Dependencies

Maven

Add the following dependencies to your pom.xml:

xml

 

<dependencies>

    <!-- Kafka Client -->

    <dependency>

        <groupId>org.apache.kafka</groupId>

        <artifactId>kafka-clients</artifactId>

        <version>3.5.1</version>

    </dependency>

    <!-- Protocol Buffers -->

    <dependency>

        <groupId>com.google.protobuf</groupId>

        <artifactId>protobuf-java</artifactId>

        <version>3.24.0</version>

    </dependency>

</dependencies>

Gradle

Add the following dependencies to your build.gradle file:

gradle

 

dependencies {

    implementation 'org.apache.kafka:kafka-clients:3.5.1'

    implementation 'com.google.protobuf:protobuf-java:3.24.0'

}

---

Step 4: Configure Kafka Properties

Update the Kafka properties in the Java code as per your setup:

• Bootstrap servers: Replace with your Kafka brokers.
• Authentication: Configure SASL_PLAINTEXT, username, and password.

Example:

java

 

props.setProperty("bootstrap.servers", "broker.lt.use1.:9094");

props.setProperty("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"SCRAM\" password=\"password\";");

---

Step 5: Build the Project

Maven

bash

 

mvn clean install

Gradle

bash

 

gradle build

---

Step 6: Run the Application

Using Maven:

bash

 

mvn exec:java -Dexec.mainClass="KafkaProtobufConsumer"

Using Gradle:

bash

 

gradle run

---

Expected Output

1. The application connects to the Kafka topic specified in the code.
2. It continuously polls messages from the topic.
3. For each message, it deserializes the Protobuf payload and prints the details.

Example output:

css

 

Received message: { sourceTime: 123456789, tradeId: "T12345" }

Received message: { sourceTime: 987654321, tradeId: "T54321" }

---

Troubleshooting

1. Error: protoc not found
• Ensure protoc is installed and available in your PATH.
• Verify with protoc --version.
2. Dependency errors:
• Run mvn dependency:resolve (Maven) or gradle dependencies (Gradle) to verify dependencies.
3. Connection issues:
• Check Kafka broker details and ensure your system has network access to the brokers.
4. Protobuf deserialization fails:
• Ensure the .proto file used to generate the Java classes matches the producer's .proto.