Configure Debezium Server

This reference document is following the original Debezium documentation. It explains how to use Debezium Server’s configuration file named application.properties to configure Debezium Server. You will find similar explanations inside the application.properties file generated by the scaffold command:

redis-di scaffold --db-type cassandra|mysql|oracle|postgresql|sqlserver --dir <PATH_TO_DIR>

Target (sink) connector

The target connector is Redis.

Basic configuration

debezium.sink.type=redis
debezium.sink.redis.address=<REDIS_DI_BDB_HOST>:12001
debezium.sink.redis.password=<REDIS_DI_PASSWORD>

Preventing data loss

In order to prevent data loss, the Debezium Redis Sink Connector needs to be configured to wait for write acknowledgment from the RDI database replica shard.

Use the following property to achieve that:

debezium.sink.redis.wait.retry.enabled=true

In addition you can configure the timeout on waiting for the replica shard acknowledgment and the delay between write retries (both 1000 milliseconds by default). See the reference section of this document for full reference.

Redis data integration configuration reference

Notes

• When using Redis to store schema history and offsets, the values of the properties debezium.source.offset.storage.redis.* and debezium.source.schema.history.internal.redis.* will be inherited from the corresponding debezium.sink.redis.* properties.
• In case you would like to override any of these defaults inherited from the sink, add them explicitly as debezium.source.offset.storage.redis.* and/or debezium.source.schema.history.internal.redis.* properties.

Source (database) connector

The source connector depends on the database you get data from. The basic configurations are the same for all database type except for the connector class.

Essential source properties

Note: Add debezium.source. prefix to the listed properties when using in application.properties.

Advanced source properties

Note: Add debezium.source. prefix to the listed properties when using in application.properties.

Debezium connectors for various databases

For additional properties consult Debezium documentation for a specific connector:

• Cassandra
• MongoDB
• MySQL/MariaDB
• Oracle
• PostgreSQL
• SQL Server

Configuring initial snapshot without filtering queries (relevant for MySQL, Oracle, PostgreSQL and SQLServer)

Tables to be included in the initial snapshot require the property debezium.source.table.include.list. They should be specified as a comma-separated list of fully-qualified table names.

Using queries in initial snapshot (relevant for MySQL, Oracle, PostgreSQL and SQLServer)

• In case you want a snapshot to include only a subset of the rows in a table, you need to add the property debezium.source.snapshot.select.statement.overrides and add a comma-separated list of fully-qualified table names. The list should include every table for which you want to add a SELECT statement.

• For each table in the list above, add a further configuration property that specifies the SELECT statement for the connector to run on the table when it takes a snapshot.

  The specified SELECT statement determines the subset of table rows to include in the snapshot.

  Use the following format to specify the name of this SELECT statement property:

  • Oracle, SQLServer, PostrgreSQL: snapshot.select.statement.overrides.<databaseName>.<tableName>
  • MySQL: snapshot.select.statement.overrides.<schemaName>.<tableName>

• Add a comma-separated list of fully-qualified column names that are included in the SELECT statement: debezium.source.column.include.list = <databaseName>.<tableName>.<columnName1>,<databaseName>.<tableName>.<columnName2>,<databaseName>.<tableName>.<columnName3>...

• In case you want to include all the table columns in the SELECT statement, you can use regular expression in the form of <databaseName>.<tableName>.* instead of adding each one of the table columns to debezium.source.column.include.list property.

NOTE: Add all tables as a comma-separated list of fully-qualified table names to the property debezium.source.table.include.list.

Example

To select the columns CustomerId, FirstName and LastName from customer table and join it with invoice table in order to get customers with total invoices greater than 8000, we need to add the following properties to the application.properties file:

debezium.source.table.include.list = chinook.customer

debezium.source.column.include.list = chinook.customer.CustomerID,chinook.customer.FirstName,chinook.customer.LastName

debezium.source.snapshot.select.statement.overrides=chinook.customer

debezium.source.snapshot.select.statement.overrides.chinook.customer = SELECT c.CustomerId, c.FirstName, c.LastName \
FROM chinook.customer c INNER JOIN chinook.invoice inv \
ON c.CustomerId = inv.CustomerId  \
WHERE inv.total > 8000

Form custom message key(s) for change event records

• By default, Debezium uses the primary key column(s) of a table as the message key for records that it emits. In place of the default, or to specify a key for tables that lack a primary key, you can configure custom message keys based on one or more columns.

• To establish a custom message key for a table, list the table followed by the column to use as the message key. Each list entry takes the following format:

  debezium.source.message.key.columns=<databaseName>.<tableName>:<columnName>
  

• To base a table key on multiple column names, insert commas between the columns names:

  debezium.source.message.key.columns=<databaseName>.<tableName>:<columnName1>,<columnName2>...
  

• The property can include entries for multiple tables. Use a semicolon to separate table entries in the list:

  debezium.source.message.key.columns=<databaseName>.<tableName1>:<columnName1>,<columnName2>;<databaseName>:<tableName>:<columnName1>,<columnName2>
  

Note: In case the property column.include.list is defined in your application.properties file, make sure it includes all the column names that are specified in the property message.key.columns.

Fully-qualified table name

In this document we refer to the fully-qualified table name as <databaseName>.<tableName>. This format is for MySQL database. For Oracle, SQLServer and Postgresql databases use <schemaName>.<tableName> instead.

Notes

• You can specify the fully-qualified table name <databaseName>.<tableName> as a regular expression instead of providing the full name of the databaseName and tableName.
• There is no limit to the number of columns that can be used to create custom message keys. However, it’s best to use the minimum required number of columns to specify a unique key.

Examples

• The primary key of the tables customer and employee is CustomerID.

  To establish custom messages keys based on FirstName and LastName for the tables customer and employee, add the following property to the application.properties file:

  debezium.source.database.message.key.columns=chinook.customer:FirstName,LastName;chinook.employee:FirstName,LastName
  

• To specify the columns FirstName and LastName as the message keys for the table chinook.customer in any database add the following property to the application.properties file:

  chinook.customer:FirstName,LastName;(.*).employee:FirstName,LastName
  

Configuring debezium connector to fetch source database secrets from secret store

Providing Source Database password in clear text is not an acceptable option. Fortunately, Debezium supports fetching secrets from environment variables or file system. This is done using a Quarkus feature that provides property expressions expansion on configuration values.

An expression string is a mix of plain strings and expression segments, which are wrapped by the sequence ${…​}.

Using Environment Variables

Property Expressions also work with environment variables, for example:

...
database.password="${MYSQL_PASSWORD}"
...

Getting secrets from file system

In addition, Debezium application.properties supports the ${file:path:key} variable syntax - the path is the path to the file and the key is the property key.

The file storing the secrets should be in a format of a single key=value per line, for example:

...
debezium.source.config.providers=file
debezium.source.config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider
database.password="${file:/tmp/debezium/secrets.txt:db-password}"
...