The goal of Redis Data Integration Benchmark Tool is to produce INSERT statements on the source database and calculates the lag between the time the key is inserted into the source DB and the time at which the key is stored in the target Redis DB.

Change record calculated latency

The databases that are supported by this tool are:

  • MySQL
  • Oracle
  • PostgreSQL
  • SQLServer

Prerequisites

  • An existing Redis Enterprise cluster version >= 6.2.
  • RedisGears >= 1.2.5 installed on the cluster.
  • A Source DB Server with Bin Log enabled. The DBs that are supported by this tool are : MySQL, Oracle, PostgreSQL and SQLServer
  • A target Redis DB.
  • A Redis Data Integration DB which is connected to the target Redis DB.
  • Debezium Server up and running and connected to the source DB and to the Redis Data Integration DB.
  • Oracle Instant Client installed and configured for Oracle source DB. In case it’s missing, follow this guide to install.

Redis Data Integration Benchmark requires connectivity to the following endpoints:

  • Redis Enterprise cluster admin API (port 9443).
  • Redis Data Integration DB (default port 12001).
  • The target Redis DB (to where Redis Data Integration is writing).
  • The source DB to which data will be written.

Installation

Download

Ubuntu 20.04

wget https://qa-onprem.s3.amazonaws.com/redis-di/latest/redis-di-benchmark-ubuntu20.04-latest.tar.gz -O /tmp/redis-di-benchmark.tar.gz

Ubuntu 18.04

wget https://qa-onprem.s3.amazonaws.com/redis-di/latest/redis-di-benchmark-ubuntu18.04-latest.tar.gz -O /tmp/redis-di-benchmark.tar.gz

RHEL 8

wget https://qa-onprem.s3.amazonaws.com/redis-di/latest/redis-di-benchmark-rhel8-latest.tar.gz -O /tmp/redis-di-benchmark.tar.gz

RHEL 7

wget https://qa-onprem.s3.amazonaws.com/redis-di/latest/redis-di-benchmark-rhel7-latest.tar.gz -O /tmp/redis-di-benchmark.tar.gz

Install Redis Data Integration Benchmark Tool

Unpack the downloaded redis-di-benchmark.tar.gz into /tmp directory:

cd /tmp
sudo tar xvf /tmp/redis-di-benchmark.tar.gz -C /usr/local/bin/

Note: Non-root users should unpack to a directory with write permission and run redis-di-benchmark directly from it.

Validate the install

Verify that redis-di-benchmark is installed by running:

redis-di-benchmark --help

Usage: redis_di_benchmark [OPTIONS] COMMAND [ARGS]...

Options:
--help Show this message and exit.

Commands:
bench

Running Redis Data Integration Benchmark Tool

Arguments

To get the list of arguments, run the following command on your terminal:

redis_di_benchmark.py bench --help
Argument Description Default Comments
rc-host Host/IP of RDI database localhost
rc-port Port of the RDI database 12001
rc-password Password for RDI database
cluster-api-port API port of Redis Enterprise cluster 9443
cluster-host Host/IP of Redis Enterprise cluster localhost
cluster-user Redis Enterprise cluster username
cluster-password Password for the Redis Enterprise cluster username
source-db-type Type of Database (mysql|oracle|postgresql|sqlserver|)
source-db-host Host/IP of the source database localhost
source-db-port Port of the source database
source-db-name The name of the database from which to stream the changes. Do not pass for Oracle DB
source-db-schema Name of the schema from which to the stream the changes- Do not pass for MySQL DB
source-db-user Username to use when connecting to the source database
source-db-password Password to use when connecting to the source database
target-db-host Host/IP of Redis target DB
target-db-port Port of Redis target DB
number-cdc-events Number of INSERT statements the tool generates per each table 50000
number-of-fields Number of fields per each table 5
no-bulk Single INSERT into the source DB in each transaction True
stream-name-prefix The prefix with which the name of the stream(s) will be created data:<serverName> The value of <serverName> should be taken from the property debezium.source.database.server.name from Debezium’s application.properties file

Run redis-di-benchmark Tool

From your current directory, run:

redis-di-benchmark bench

Note: Non-root users should run the tool from the directory where the file redis-di-benchmark.tar.gz was extracted.

Output

For each key, Redis Data Integration Benchmark Tool calculates the difference between the arrival time of the key to Redis target DB and the time it was inserted to the source database. When all the keys arrive to Redis target database, it displays the following statistics:

Statistics with input argument no-bulk = True

End-to-end latency:
	Minimum latency: 0.885s
	Maximum latency: 6.343s
	Mean latency: 3.822s
	Median latency:  3.809s
	Standard deviation:  1.046s

Statistics with input argument no-bulk = False:

End-to-end latency:
	Minimum latency: 1.670s
	Maximum latency: 3.875s
	Mean latency: 2.743s
	Median latency:  2.739s
	Standard deviation:  0.877s
Throughput:
	Number of tables: 1
	Number of fields per each table: 5
	Size of field 10B
	Number of row in each table: 5000
	Number of events per second: 1233 eps

Notes

  • When no-bulk = True, the tool inserts one record at a time.
  • When no-bulk = False, the tool inserts 5000 records at a time.

Install Oracle Instant Client

  • Create a dedicated Oracle folder:

    sudo mkdir /opt/oracle

  • Download and unzip Oracle Instant Client:

    cd opt/oracle
sudo wget https://download.oracle.com/otn_software/linux/instantclient/214000/instantclient-basic-linux.x64-21.4.0.0.0dbru.zip
sudo unzip instantclient-basic-linux.x64-21.4.0.0.0dbru

  • Install libaio:

    • Ubuntu 18.04, Ubuntu 20.04:

      sudo apt update
sudo apt install libaio1

    • RHEL 7, RHEL 8:

      sudo yum update
sudo yum install libaio

  • Install unixODBC:

    • Ubuntu 18.04, Ubuntu 20.04:

      sudo apt install unixodbc

    • RHEL 7, RHEL 8:

      sudo yum install unixODBC

  • Add and link system configuration for Oracle Instant Client path:

    sudo sh -c "echo /opt/oracle/instantclient_21_4 > /etc/ld.so.conf.d/oracle-instantclient.conf"
sudo ldconfig