The key functionality that RDI performs is mapping the data coming from Debezium Server (representing a Source Database row data or row state change) into a Redis key with a value of Hash or JSON. There are two types of Data Transformations in RDI:

  • By default, each source row is converted into one Hash or one JSON key in Redis. This conversion uses the Debezium schema-based conversion: the incoming data includes the schema and RDI uses a set of handlers to automatically convert each source column to a Redis Hash field or JSON type based on the Debezium type in the schema. See Data type conversion for a full reference on these conversions.

  • If the user wants to add or modify this default mapping, RDI provides Declarative Data Transformations. These transformations are represented in YAML files. Each file contains a Job, a set of transformations per source table. More about Declarative Transformations.

Data transformation flow

More info

    Data transformation pipeline

    Learn how to transform data to Redis types

    Data type handling

    Describes how relational data types are converted to Redis data types

    Data denormalization

    Learn denormalization strategies

    Write-behind foreach example The foreach section is used in order to explode a list of objects or arrays to rows in a selected target. The foreach expression is structured as <field_name>:. The following example uses the add_fieldtransformation to prepare the input JSON to the desired structure. Then, it appliesforeachto write eachorderobject as a relational database record usingkeysandmapping. In this example, the JMESPathfunctionto_stringis used to flatten an array of objectsspecs` to a string.