Enforce Record Size

The EnforceRecordSize SMT estimates the serialized size of each record and, if it exceeds the configured maximum, truncates eligible string and bytes columns proportionally to bring the record within the size limit.

Some downstream systems, such as Apache Kafka brokers or sink connectors, impose a maximum message size limit. When Debezium captures a change event for a row that contains large string or bytes columns, the serialized record can exceed that limit, causing the connector to fail.

The SMT truncates larger columns aggressively to comply with the threshold. If does not modify columns if their sizes are at or below the configured threshold.

Example: Basic configuration

You apply the SMT to a connector by adding its details to the connector’s transforms configuration. The following example shows a configuration for the SMT that limits records to 1 MB:

transforms=EnforceRecordSize
transforms.EnforceRecordSize.type=io.debezium.transforms.EnforceRecordSize
transforms.EnforceRecordSize.max.bytes=1048576

When this configuration is active, if the estimated serialized size of a record exceeds 1,048,576 bytes (1 MB), its large string and bytes fields are truncated proportionally until the record conforms to the configured limit.

How the SMT estimates record size

The SMT uses an approximate size calculation that examines the key, value, headers, and topic of the record. For string fields, it uses String.length() as a constant-time proxy for byte size. The string size calculation assumes one byte per character, which understates multi-byte UTF-8 content but avoids the performance cost of encoding each string.

When the estimated size (adjusted by the optional compression.ratio) exceeds max.bytes, the SMT identifies all string and bytes fields in the before and after sections of the envelope that are larger than min.field.size. It then distributes the excess bytes proportionally across those fields, truncating larger fields more than smaller fields.

Example: Configuration with compression ratio

If your Kafka producer uses compression, for example, LZ4 or Snappy, the actual size of a record as transmitted over the network is smaller than its raw serialized size. You can account for the use of compression by setting a compression.ratio in the configuration with a value of less than 1.0.

For example, if Kafka producer metrics show an average compression rate of 0.6, set the compression.ratio as in the following example:

transforms=EnforceRecordSize
transforms.EnforceRecordSize.type=io.debezium.transforms.EnforceRecordSize
transforms.EnforceRecordSize.max.bytes=1048576
transforms.EnforceRecordSize.compression.ratio=0.6

With this configuration, the SMT multiplies the estimated raw size by 0.6 before comparing it to the 1 MB limit. As a result, the SMT only truncates columns if the compressed size would exceed the limit, avoiding unnecessary truncation.

You can determine the effective compression ratio by examining Kafka producer metrics: kafka.producer:type=producer-metrics,client-id=<id>/compression-rate-avg.

Example: Protecting small fields from truncation

By default, the SMT only truncates fields larger than 25,000 bytes (25 KB). By acting only on these large fields, the SMT protects small fields, which are unlikely to contribute meaningfully to size reduction, from being truncated.

You can modify the default threshold by setting the min.field.size attributes in the configuration, as in the following example:

transforms=EnforceRecordSize
transforms.EnforceRecordSize.type=io.debezium.transforms.EnforceRecordSize
transforms.EnforceRecordSize.max.bytes=1048576
transforms.EnforceRecordSize.min.field.size=10000

If you set min.field.size=10000, the SMT only truncates fields larger than 10 KB. To make all string and bytes fields eligible for truncation, set min.field.size=0.

Behavior details

The enforce record size SMT evaluates only Debezium change event records and applies truncation to the before and after fields of the event envelope. Size estimates are approximate; the actual serialized size depends on the converter you use.

During processing, the enforce record size transformation applies the following constraints and behaviors.

• The SMT processes only records whose value contains a Debezium change event envelope, that is, a Struct with before and after fields. It passes non-envelope records, such as schema change events or tombstones, unchanged.
• The SMT applies truncation to both the before and after sections of the envelope and distributes the excess between the two sections in proportion to their truncatable content.
• The SMT does not guarantee that the final serialized record is exactly at or below max.bytes, because it estimates size approximately and the actual serialized size depends on the converter (JSON, Avro, or Protobuf) and its associated overhead. To provide a safety margin, configure a conservative max.bytes value or adjust the compression.ratio setting.

Configuration options

The following table lists the configuration options that you can set for the enforce record size SMT.

Table 1. Enforce record size SMT (EnforceRecordSize) configuration options