Convert between geometry formats

The GeometryFormatTransformer single message transform (SMT) converts geometry values in change events between WKB and EWKB format to ensure that target systems receive spatial data in the most compatible format. The transformation supports all geometry types (Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, and GeometryCollection), and is particularly useful when integrating with systems that require a specific format. By optimizing the data format, you can prevent data processing failures and help to ensure efficient data flows across the heterogeneous systems in your data pipeline.

About spatial data formats

WKB (Well-Known Binary) and EWKB (Extended Well-Known Binary) are two common formats for representing spatial data. WKB is the standard Open Geospatial Consortium (OGC) format for representing geometry data in binary form. EWKB is a PostGIS extension that adds SRID (Spatial Reference System Identifier) information to the WKB structure. An SRID is a unique identifier that specifies the coordinate reference system that the geometry uses. It defines how coordinates are interpreted on Earth’s surface, which is essential for accurate spatial operations like distance calculations and coordinate transformations.

WKB format stores SRIDs separately, whereas EWKB, as used in the PostgreSQL PostGIS spatial extension, encodes the SRID directly within the binary geometry data itself. The SMT ensures format compatibility between different database systems (for example, PostgreSQL/PostGIS uses EWKB, while MySQL, SQL Server, and Oracle use WKB) Apply the SMT to source connectors to transform events before Kafka Connect writes them to Kafka, or to sink connectors to transform data before the sink connector writes it to the target system.

The GeometryFormatTransformer SMT uses the 'io.debezium.data.Geometry' logical type to identify geometry fields.

When to use the GeometryFormatTransformer SMT

The GeometryFormatTransformer SMT helps to ensure data compatibility for consumers that require different spatial data formats. When you know that a downstream system will reject the source data format or fail to parse it correctly, apply the GeometryFormatTransformer to ensure that the consumer receives data in a format that it can process.

You can apply the transformation to either a Debezium source connector or a sink connector. When configured on a Debezium source connector, the SMT standardizes the format of spatial data in change event records before the connector writes them to Kafka.

When configured on a sink connector, the SMT converts the format of spatial data in change event records after they are consumed from a Kafka topic and before the connector writes them to the downstream system. To function correctly, the records must conform to the Debezium change event structure, as produced by Debezium source connectors.

Apply the GeometryFormatTransformer in the following situations:

Streaming to PostGIS or spatial databases

When you have source data in WKB format but a target system uses PostGIS or another spatial database that requires SRID (Spatial Reference System Identifier) information, use the SMT to convert to EWKB format. Without SRID data, PostGIS cannot accurately perform spatial operations like distance calculations, coordinate transformations, or spatial indexing. The SMT enables Debezium to move spatial data between databases that use different storage conventions while preserving spatial coordinate references.

Integrating with legacy GIS applications

When you have source data in EWKB format and downstream systems run older GIS tools or applications that cannot parse SRID extensions, use the SMT to convert data to the simpler WKB format.

Building multi-target data pipelines

When you route spatial data to multiple downstream systems, run the SMT to customize data to ensure that each system receives data in its required format.

Preventing data processing failures

When you encounter geometry parsing errors, null values, or data type mismatches in your downstream systems, the cause is often format incompatibility. Converting data to a more consumable format can help to resolve errors and optimize data flow.

Apply the SMT to convert between WKB and EWKB formats

To convert the format of WKB or EWKB data in a data source, add the GeometryFormatTransformer SMT to the connector configuration, and specify the target format. Based on the configuration, the transformer intelligently detects the source format and converts it accordingly, ensuring that the output format matches the target system’s requirements.

You can add the SMT without explicitly specifying the target format, to configure it to convert data from EWKB format to WKB format. You can modify the default target to convert WKB data to EWKB format. To preserve SRID information (convert to EWKB), you must explicitly set the format to EWKB.

Procedure

1. To configure the default behavior of the transformation (convert from EWKB format to WKB format), add the SMT to the connector configuration without specifying the format option, as in the following example:

   transforms=convertGeometryFormat
   transforms.convertGeometryFormat.type=io.debezium.transforms.GeometryFormatTransformer

2. To configure the transformation to convert from WKB format to EWKB format, specify EWKB as the target format in your connector configuration, as shown in the following example:

transforms=convertGeometryFormat
transforms.convertGeometryFormat.type=io.debezium.transforms.GeometryFormatTransformer
transforms.convertGeometryFormat.format=EWKB

Configuration options

The following table lists the configuration options that you can use with the GeometryFormatTransformer SMT.

Table 1. GeometryFormatTransformer SMT configuration options