When running Redis Data Integration in a Kubernetes environment, we recommend creating a ConfigMap and adding the RDI CLI as a pod in the cluster.

Throughout the document, the snippets make use of the Kubernetes kubectl tool. All examples can be replaced with oc when working in an OpenShift environment.

Prerequisites

Install RDI CLI

There are two options for installing the RDI CLI in an Kubernetes environment:

Create a new RDI database

  • Run:

    kubectl port-forward service/rec 9443:9443

    *If the Redis Enterprise Cluster service is not named rec (the default), make sure to forward to the correct service name.

  • Run create command to set up a new Redis Data Integration database instance within an existing Redis Enterprise Cluster:

    redis-di create --cluster-host localhost --no-configure

  • Next we need to run the command port-forward command to be able to connect to Redis Data Integration inside the Kubernetes cluster:

    kubectl port-forward service/<REDIS_DI_SERVICE_NAME> <REDIS_DI_PORT>:<REDIS_DI_PORT>

    Replace the values for the port-forward with the service name and port of the Redis Data Integration BDB that was created using the create command.

  • Configure Redis Data Integration:

    redis-di configure

    You should get a message - “Successfully configured redis-di instance on port <REDIS_DI_PORT>”

The create command will create a BDB named redis-di-1 in your cluster. You will need to use a privileged Redis Enterprise user that has the permissions to create a BDB and to register Gears recipes, to run it.

Create configuration file for Redis Data Integration

Run redis-di scaffold --db-type <{{param rdi_db_types}}> --dir <PATH_TO_DIR>. Edit the file config.yaml which is located under the directory to point to the correct Redis Target database settings:

connections:
  # Redis target DB connection details
  target:
    host: <REDIS_TARGET_DB_SERVICE_NAME>
    port: <REDIS_TARGET_DB_PORT>
    #password: <REDIS_TARGET_DB_PASSWORD>
    # In case of target DB password stored in a secret and mounted as an env variable
    # available to the Redis Enterprise Cluster pod
    #password: ${env:redis-target-password}

Run redis-di deploy command to deploy the configuration in the config.yaml file to the remote RDI database.

For more information, see config.yaml reference.

Validate the deploy

Run redis-di status to check the status of the installation.

Install RDI CLI on Kubernetes Cluster

Add CLI pod

cat << EOF > /tmp/redis-di-cli-pod.yml
apiVersion: v1
kind: Pod
metadata:
  name: redis-di-cli
  labels:
    app: redis-di-cli
spec:
  containers:
    - name: redis-di-cli
      image: docker.io/redislabs/redis-di-cli
      volumeMounts:
      - name: config-volume
        mountPath: /app
      - name: jobs-volume
        mountPath: /app/jobs
  volumes:
    - name: config-volume
      configMap:
        name: redis-di-config
        optional: true
    - name: jobs-volume
      configMap:
        name: redis-di-jobs
        optional: true
EOF
kubectl apply -f /tmp/redis-di-cli-pod.yml

To run the CLI commands, use:

kubectl exec -it pod/redis-di-cli -- redis-di

Create configuration file for Redis Data Integration

  • Run the following command to create the configuration file for Redis Data Integration:

    kubectl exec -it pod/redis-di-cli -- redis-di scaffold --db-type <{{param  rdi_db_types}}> --preview config.yaml > config.yaml

  • Edit the file config.yaml to point to the correct Redis Target database settings.

For config.yaml reference see complete settings.

Create ConfigMap for Redis Data Integration

Run the following command to create the ConfigMap for Redis Data Integration:

kubectl create configmap redis-di-config --from-file=config.yaml

Create new RDI database

Run create command to set up a new Redis Data Integration database instance within an existing Redis Enterprise Cluster:

kubectl exec -it pod/redis-di-cli -- redis-di create

The create command will create a BDB named redis-di-1 in your cluster. You will need to use a privileged Redis Enterprise user that has the permissions to create a BDB and to register Gears recipes, to run it.

Deploy configuration

Run deploy command to deploy the configuration in the ConfigMap to the remote redis-di instance:

kubectl exec -it pod/redis-di-cli -- redis-di deploy

Read more about deploying data transformation jobs when the RDI CLI is deployed as a Kubernetes pod here.

Validate the installation

Run kubectl exec -it pod/redis-di-cli -- redis-di status to check the status of the installation.

Note that it is OK to see the warning of “No streams found” since we have not yet set up a Debezium source connector. We will do this in the next step.

Install the Debezium Server

Create configuration file for Redis Data Integration

  • Run the following command to create the configuration file for Debezium Server:

    kubectl exec -it pod/redis-di-cli -- redis-di scaffold --db-type <{{param  rdi_db_types}}> --preview debezium/application.properties > application.properties

  • Edit the file application.properties and replace the values for the debezium.sink with the service name and credentials of the Redis Data Integration BDB that was created using the create command.

    Note: make sure to reference the Service name that was created by the service rigger.

For a full list of configuration options and classname of each connector see the Debezium documentation

Create a ConfigMap for Debezium Server

Run the following command to create the ConfigMap for Debezium Server:

kubectl create configmap debezium-config --from-file=application.properties

Create the Debezium Server Pod

cat << EOF > /tmp/debezium-server-pod.yml
apiVersion: v1
kind: Pod
metadata:
  name: debezium-server
  labels:
    app: debezium-server
spec:
  containers:
    - name: debezium-server
      image: docker.io/debezium/server
      livenessProbe:
        httpGet:
            path: /q/health/live
            port: 8088
      readinessProbe:
        httpGet:
            path: /q/health/ready
            port: 8088
      volumeMounts:
      - name: config-volume
        mountPath: /debezium/conf
  volumes:
    - name: config-volume
      configMap:
        name: debezium-config
EOF
kubectl apply -f /tmp/debezium-server-pod.yml

Optional: Set up an example database

The Debezium project has created preconfigured example databases for many relational database.

To set up a Postgres example database:

cat << EOF > /tmp/example-postgres.yml
apiVersion: v1
kind: Pod
metadata:
  name: example-postgres
  labels:
    app: postgres
spec:
  containers:
    - name: example-postgres
      image: docker.io/debezium/example-postgres
      ports:
      - containerPort: 5432
      env:
      - name: POSTGRES_USER
        value: "postgres"
      - name: POSTGRES_PASSWORD
        value: "postgres"

---

apiVersion: v1
kind: Service
metadata:
  name: example-postgres
  labels:
    app: postgres
spec:
  type: NodePort
  ports:
  - port: 5432
  selector:
    app: postgres
EOF
kubectl apply -f /tmp/example-postgres.yml