This guide will take you through the creation of a write-behind pipeline.

Prerequisites

Create a new RDI database

Run create command to set up a new Redis Data Integration database instance within an existing Redis Enterprise Cluster:

redis-di create

The create command will create a BDB named redis-di-<ID> in your cluster. You will need to use a privileged Redis Enterprise user that has the permissions to create a BDB and to register Gears recipes, to run it.

Note: This command requires credentials of a Redis Enterprise cluster user with either DB Member, Cluster Member or Cluster Admin roles, read more about cluster roles.

Scaffold configuration files

Reference documentation

Run scaffold command to generate configuration files for Redis Data Integration and Debezium Redis Sink Connector:

redis-di scaffold --db-type <cassandra|mysql|oracle|postgresql|sqlserver> --dir <PATH_TO_DIR>

The following files will be created in the provided directory:

├── debezium
│   └── application.properties
├── jobs
│   └── README.md
└── config.yaml

Folder Structure

  • config.yaml - Redis Data Integration configuration file (definitions of target database, applier, and so on).

  • debezium/application.properties - Debezium Server configuration file.

  • jobs - Data transformation jobs. For more information, see Data transformation pipeline.

Update Redis target connection details

config.yaml diagram

Edit the config.yaml configuration file. This file holds the connection details of the target Redis instance and Applier settings.

Update the connection/target details to match the target database settings. See configuration reference for all available settings.

Preventing data loss at Redis target DB

In order to prevent Redis Target DB from losing data, configure RDI to wait for replica shard write acknowledgment. This can be done by adding the following lines in the applier section of config.yaml:

applier:
  wait_enabled: true
  retry_on_replica_failure: true

Deploy configuration

Run deploy command to deploy the local configuration to the remote RDI database:

redis-di deploy

Note: If you are specifying TLS key, cert and cacert locations in config.yaml or you use one of the file patterns ${file:<location>} ${file:<location>:<property-name>} for the password property, make sure these files exist on the Redis Enterprise nodes that host RDI shards.

Validate the install

Run redis-di status to check the status of the installation.

Note that it is OK to see the warning of “No streams found” since we have not yet set up a Debezium source connector. We will do this in the next step.

Install the Debezium Server

Configure Debezium Server’s application.properties

application.properties Diagram

Edit debezium/application.properties file created under your project directory (created by the scaffold command described above).

You can read the Debezium Server Configuration Guide for the key configuration properties reference.

Run Debezium Server

Follow Debezium Server Deployment Guide to deploy and run Debezium Server in either containerized and non-containerized deployment modes of your choice.

Quick database setup using Debezium example database

In case you want a quick setup of a preconfigured database to experiment with Redis Data Integration, you can use one of the Debezium example databases.

To set up a Postgres example database:

docker run -it --name example-postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 debezium/example-postgres

Redis Data Integration Configuration File (config.yaml)

Reference Guide

See Redis Data Integration configuration file reference.

Substitutions

  • ${env:var} - Environment variables. The env variables will be prefixed with REDIS_DI_ if not already prefixed that way.

  • ${file:<FILE_NAME>:<PROPERTY_NAME>} - Specific property in properties file.

  • ${file:<FILE_NAME>} - Reading the whole file.

  • ${secrets:secret_name} - Secrets: To set the secret use the redis-di set-secret CLI command. Currently only supports the secret named target-password.

Distributed installation

For information about a distributed installation (multi VM/Multi pod), see Distributed installation.

Installing from a Python package (Whl File)

Note: RDI CLI requires Python version 3.7 and up.

  • Set up a new virtual environment for Python:

    python3.8 -m venv venv  # replace with python3.9 if in use

  • Activate virtual env:

    source venv/bin/activate

  • Upgrade Pip:

    pip install --upgrade pip

  • Install RDI CLI:

    pip3 install https://qa-onprem.s3.amazonaws.com/redis-di/latest/redis_di_cli-latest-py3-none-any.whl

Upgrading

To learn how you can upgrade RDI CLI, RDI engine, and RedisGears see the Upgrade section.

Troubleshooting

See the troubleshooting guide.