Ingest quickstart
This guide will take you through the creation of a write-behind pipeline.
Prerequisites
- Install RDI CLI.
- An existing Redis Enterprise cluster version >= 6.2.
- RedisGears >= 1.2.5 installed on the cluster. In case it’s missing, see Install RedisGears for Redis Data Integration to install.
- A target Redis database (can be added after installation).
Create a new RDI database
Run create
command to set up a new Redis Data Integration database instance within an existing Redis Enterprise Cluster:
redis-di create
The create
command will create a BDB named redis-di-<ID>
in your cluster. You will need to use a privileged Redis Enterprise user that has the permissions to create a BDB and to register Gears recipes, to run it.
Note: This command requires credentials of a Redis Enterprise cluster user with either
DB Member
,Cluster Member
orCluster Admin
roles, read more about cluster roles.
Scaffold configuration files
Run scaffold
command to generate configuration files for Redis Data Integration and Debezium Redis Sink Connector:
redis-di scaffold --db-type <cassandra|mysql|oracle|postgresql|sqlserver> --dir <PATH_TO_DIR>
The following files will be created in the provided directory:
├── debezium
│ └── application.properties
├── jobs
│ └── README.md
└── config.yaml
Folder Structure
-
config.yaml
- Redis Data Integration configuration file (definitions of target database, applier, and so on). -
debezium/application.properties
- Debezium Server configuration file. -
jobs
- Data transformation jobs. For more information, see Data transformation pipeline.
Update Redis target connection details
Edit the config.yaml
configuration file. This file holds the connection details of the target Redis instance and Applier settings.
Update the connection/target
details to match the target database settings. See configuration reference for all available settings.
Preventing data loss at Redis target DB
In order to prevent Redis Target DB from losing data, configure RDI to wait for replica shard write acknowledgment. This can be done by adding the following lines in the applier
section of config.yaml
:
applier:
wait_enabled: true
retry_on_replica_failure: true
Deploy configuration
Run deploy command to deploy the local configuration to the remote RDI database:
redis-di deploy
Note: If you are specifying TLS
key
,cert
andcacert
locations inconfig.yaml
or you use one of the file patterns${file:<location>}
${file:<location>:<property-name>}
for thepassword
property, make sure these files exist on the Redis Enterprise nodes that host RDI shards.
Validate the install
Run redis-di status
to check the status of the installation.
Note that it is OK to see the warning of “No streams found” since we have not yet set up a Debezium source connector. We will do this in the next step.
Install the Debezium Server
Configure Debezium Server’s application.properties
Edit debezium/application.properties
file created under your project directory (created by the scaffold
command described above).
You can read the Debezium Server Configuration Guide for the key configuration properties reference.
Run Debezium Server
Follow Debezium Server Deployment Guide to deploy and run Debezium Server in either containerized and non-containerized deployment modes of your choice.
Quick database setup using Debezium example database
In case you want a quick setup of a preconfigured database to experiment with Redis Data Integration, you can use one of the Debezium example databases.
To set up a Postgres example database:
docker run -it --name example-postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 debezium/example-postgres
Redis Data Integration Configuration File (config.yaml)
Reference Guide
See Redis Data Integration configuration file reference.
Substitutions
-
${env:var}
- Environment variables. The env variables will be prefixed withREDIS_DI_
if not already prefixed that way. -
${file:<FILE_NAME>:<PROPERTY_NAME>}
- Specific property in properties file. -
${file:<FILE_NAME>}
- Reading the whole file. -
${secrets:secret_name}
- Secrets: To set the secret use theredis-di set-secret
CLI command. Currently only supports the secret namedtarget-password
.
Distributed installation
For information about a distributed installation (multi VM/Multi pod), see Distributed installation.
Installing from a Python package (Whl File)
Note: RDI CLI requires Python version 3.7 and up.
-
Set up a new virtual environment for Python:
python3.8 -m venv venv # replace with python3.9 if in use
-
Activate virtual env:
source venv/bin/activate
-
Upgrade Pip:
pip install --upgrade pip
-
Install RDI CLI:
pip3 install https://qa-onprem.s3.amazonaws.com/redis-di/latest/redis_di_cli-latest-py3-none-any.whl
Upgrading
To learn how you can upgrade RDI CLI, RDI engine, and RedisGears see the Upgrade section.
Troubleshooting
See the troubleshooting guide.