Kafka Protobuf Consumer
This project demonstrates how to consume messages from a
Kafka topic using a Java application. The messages are serialized using
Protocol Buffers (Protobuf) and deserialized in the application.
Prerequisites
- Java
Development Kit (JDK): Install JDK 11 or later.
- Apache
Kafka: Ensure you have access to a Kafka cluster.
- Protocol
Buffers Compiler (protoc):
- Download
the correct version for your platform from the Protocol
Buffers GitHub releases.
- Add
the protoc binary to your system PATH.
- Verify
installation:
bash
protoc --version
- Maven
or Gradle: Install Maven or Gradle to manage project dependencies.
Setup Instructions
Step 1: Clone or Download the Project
bash
git clone <repository-url>
cd KafkaProtobufConsumer
Step 2: Compile Protobuf File
- Place
your .proto file (e.g., PPP_Cloud_Streaming.proto) in the src/main/proto
directory.
- Use protoc
to generate Java files:
bash
protoc --java_out=src/main/java src/main/proto/PPP_Cloud_Streaming.proto
- The
generated Java files will be placed in the src/main/java directory under
the specified package.
Step 3: Configure Project Dependencies
Maven
Add the following dependencies to your pom.xml:
xml
<dependencies>
<!-- Kafka
Client -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>3.5.1</version>
</dependency>
<!-- Protocol
Buffers -->
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.24.0</version>
</dependency>
</dependencies>
Gradle
Add the following dependencies to your build.gradle file:
gradle
dependencies {
implementation
'org.apache.kafka:kafka-clients:3.5.1'
implementation
'com.google.protobuf:protobuf-java:3.24.0'
}
Step 4: Configure Kafka Properties
Update the Kafka properties in the Java code as per your
setup:
- Bootstrap
servers: Replace with your Kafka brokers.
- Authentication:
Configure SASL_PLAINTEXT, username, and password.
Example:
java
props.setProperty("bootstrap.servers", "broker.lt.use1.:9094");
props.setProperty("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule
required username=\"SCRAM\" password=\"password\";");
Step 5: Build the Project
Maven
bash
mvn clean install
Gradle
bash
gradle build
Step 6: Run the Application
Using Maven:
bash
mvn exec:java -Dexec.mainClass="KafkaProtobufConsumer"
Using Gradle:
bash
gradle run
Expected Output
- The
application connects to the Kafka topic specified in the code.
- It
continuously polls messages from the topic.
- For
each message, it deserializes the Protobuf payload and prints the details.
Example output:
css
Received message: { sourceTime: 123456789, tradeId: "T12345"
}
Received message: { sourceTime: 987654321, tradeId: "T54321"
}
Troubleshooting
- Error:
protoc not found
- Ensure
protoc is installed and available in your PATH.
- Verify
with protoc --version.
- Dependency
errors:
- Run mvn
dependency:resolve (Maven) or gradle dependencies (Gradle) to verify
dependencies.
- Connection
issues:
- Check
Kafka broker details and ensure your system has network access to the
brokers.
- Protobuf
deserialization fails:
- Ensure
the .proto file used to generate the Java classes matches the producer's .proto.