Integrating Redpanda with Redis Using the Redis Connector

Overview

This document provides a step-by-step guide to integrating a Redpanda cluster with Redis using the Redis connector. It includes downloading, configuring, and deploying the Redis connector for Redpanda Enterprise version 24.1.

---

Prerequisites

• Redpanda Enterprise 24.1 is installed and running.

• Redis is installed and accessible.

• Kafka Connect is installed and running (if required).

---

Step 1: Set Up Redis Cluster

1. Install Redis (if not already installed):

   sudo apt update && sudo apt install redis-server -y
   

   Or for CentOS/RHEL:

   sudo yum install redis -y
   

2. Modify Redis configuration to enable clustering: Edit the Redis configuration file (/etc/redis/redis.conf or /etc/redis.conf):

   cluster-enabled yes
   cluster-config-file nodes.conf
   cluster-node-timeout 5000
   appendonly yes
   ``
   Save and exit.
   
   

3. Start Redis service:

   sudo systemctl restart redis
   sudo systemctl enable redis
   

4. Create a Redis Cluster (Assuming three Redis nodes: 6379, 6380, 6381):

   redis-cli --cluster create 127.0.0.1:6379 127.0.0.1:6380 127.0.0.1:6381 --cluster-replicas 1
   

5. Verify cluster status:

   redis-cli -c
   cluster nodes
   

---

Step 2: Verify Redpanda Connect Availability

Before proceeding, ensure that Redpanda Connect is enabled.

Check if Redpanda Connect is Enabled

rpk cluster config get redpanda.enable_connect

If the output is false, enable it with:

rpk cluster config set redpanda.enable_connect true
rpk cluster restart

Check if Redpanda Connect Worker is Running

rpk connect status

---

Step 3: Download and Install the Redis Connector (If Required Manually)

Verify if the Redis Connector is Available

Before manually downloading, check if the Redis connector is already available:

rpk connect plugins list

If redis or related connectors (redis_hash, redis_list, redis_pubsub, etc.) are listed, skip the manual installation.

Download the Redis Connector JAR

If the connector is missing, follow these steps:

1. Find the official Redis Connector JAR from Redpanda or Confluent Hub

   • Redpanda: https://docs.redpanda.com (Check the connector catalog)

   • Confluent Hub: https://www.confluent.io/hub (Search for "Redis Sink Connector")

2. Download the latest JAR file

   wget -O redis-sink-connector.jar https://repo.redpanda.com/path/to/redis-sink-connector.jar
   

   Or for Confluent’s Redis connector:

   wget -O redis-sink-connector.zip https://packages.confluent.io/maven/io/confluent/kafka-connect-redis/1.0.0/kafka-connect-redis-1.0.0.zip
   

3. Extract if necessary (for .zip files)

   unzip kafka-connect-redis-1.0.0.zip -d /opt/redpanda/connectors/
   

Install the Connector

1. Find the Kafka Connect plugin path

   rpk connect plugins path
   

   Example output:

   /opt/redpanda/connect/plugins/
   

2. Move the Redis connector JAR to the plugins directory

   mv redis-sink-connector.jar /opt/redpanda/connect/plugins/
   

3. Restart Redpanda Connect

   rpk cluster restart
   

4. Verify the connector is now available

   rpk connect plugins list
   

   The Redis connector should now be listed.

---

Step 4: Configure the Redis Connector

1. Create a configuration file (e.g., redis-connector.json)

   {
     "name": "redis-sink-connector",
     "config": {
       "connector.class": "com.redpanda.kafka.connect.redis.RedisSinkConnector",
       "tasks.max": "1",
       "topics": "your_redpanda_topic",
       "redis.uri": "redis://redpanda-redis",
       "redis.key.format": "string",
       "redis.value.format": "json",
       "key.converter": "org.apache.kafka.connect.storage.StringConverter",
       "value.converter": "org.apache.kafka.connect.json.JsonConverter",
       "value.converter.schemas.enable": "false"
     }
   }
   

   Replace your_redpanda_topic with the actual Redpanda topic name.

---

Step 5: Deploy the Redis Connector

1. Use the rpk connect CLI to add the connector

   rpk connect create redis-connector --config redis-connector.json
   

2. Verify that the connector is running

   rpk connect status redis-connector
   

3. Check logs if needed

   rpk connect logs redis-connector
   

---

Step 6: Verify Data Flow

1. Produce test data to Redpanda topic

   rpk topic produce your_redpanda_topic
   

   Enter sample data in JSON format:

   {"id": 1, "name": "test"}
   

   Press Ctrl+D to send.

2. Check if data appears in Redis Connect to Redis:

   redis-cli
   

   Run:

   KEYS *
   

   Then:

   GET your_key
   

---

Troubleshooting

• If the connector fails, check logs with rpk connect logs redis-connector.

• Ensure Redis is accessible from the Redpanda cluster.

• Verify that the Kafka Connect worker is running (rpk connect status).

---

Conclusion

Following these steps, you can successfully integrate Redpanda with Redis using the Redis connector, enabling real-time data streaming between Redpanda and Redis.

https://docs.confluent.io/kafka-connectors/redis/current/overview.html

https://www.confluent.io/hub/jcustenborder/kafka-connect-redis?session_ref=https://www.google.com/&_ga=2.154182525.454879604.1743598509-1505666821.1743598509&_gl=1*142o3k*_gcl_au*MTA0MzY4OTY5OC4xNzQzNTk4NTA4*_ga*MTUwNTY2NjgyMS4xNzQzNTk4NTA5*_ga_D2D3EGKSGD*MTc0MzU5ODUwOC4xLjEuMTc0MzU5ODUzMC4zOC4wLjA.