Transformations

Connectors can be configured with transformations to make lightweight per message modifications. Debezium provides several single message transformations (SMTs) that you can use to either modify records before they are sent to Apache Kafka (by applying them to the Debezium connectors), or when they are read from Kafka by a sink connector. Also Debezium Server supports the usage of SMTs.

The following SMTs are provided by Debezium:

Transform	Description
Topic Routing	Re-routes records to different topics based on a regular expression applied to the original topic name.
Content-Based Routing	Reroute selected events to other topics, based on the event content.
New Record State Extraction	Extracts the flat structure of field names and values from Debezium change events, facilitating sink connectors which cannot process Debezium’s complex event structure.
MongoDB New Document State Extraction	The MongoDB-specific counter-part to the New Record State Extraction SMT.
Outbox Event Router	Provides a way to safely and reliably exchange data between multiple (micro) services.
MongoDB Outbox Event Router	The MongoDB-specific counter-part to the Outbox Event Router SMT.
Message Filtering	Applies a filter to the change events emitted by the connectors, based on their content. This lets you propagate only those records that are relevant to you.
HeaderToValue	Moves or copies headers into the record value.
Partition Routing	Re-routes records to specific partitions based on configured payload fields.
Timezone Converter	Converts Debezium and Kafka Connect timestamp fields in event records to a specified timezone.
Geometry Format Transformer	Converts Debezium Geometry field types between Well-Known Binary(WKB) and Extended Well-Known Binary(EWKB) formats.
TimescaleDB Integration	Routes and enriches messages that the Debezium PostgreSQL connector captures from a TimescaleDB.
Decode Logical Decoding Message Content	Converts the binary content of logical decoding messages captured by the Debezium PostgreSQL connector into a structured form.
Enforce Record Size	Enforces a maximum record size by proportionally truncating large string and bytes columns when a change event exceeds the configured limit.
Vitess Use Local VGTID	A transformation that reduces the size of VGTIDs that the Vitess connector emits. To reduce the amount of data in the event message, the SMT writes only the VGTID for the shard in which a change occurs. The VGTIDs for other shards are removed. This transformation is designed for use only with the Debezium connector for Vitess.
Vitess Remove Field	The Vitess-specific transformation to remove fields to reduce the size of the messages written (e.g., redundant VGTID fields).
Vitess Replace Field Value	The Vitess-specific transformation to replace the value of a field to reduce the size of the messages written (e.g., redundant VGTID fields).
Vitess Filter Transaction Topic Records	A transformation that filters out all messages that the connector emits to the transaction topic (when transaction metadata is enabled). This transformation is designed for use only with the Debezium connector for Vitess.

By means of SMT Predicates you can apply any of the transformations selectively, so that it modifies only that subset of change event messages that share a common characteristic.

	Most of the above SMTs are available by default with the Debezium container image, but you need to opt-in for the scripting-based ones (Message Filtering or Content-based Routing). See the README file of the container image for more details.

Overview

Transformations

评论