Debezium Operator

Overview

The Debezium Operator is a Kubernetes operator that provides automated deployment and management of Debezium Server instances on Kubernetes and OpenShift platforms. This operator implements the standard Kubernetes operator pattern to transform user-defined DebeziumServer custom resources into running change data capture (CDC) pipelines that stream database changes to various sink systems.

Deploy Debezium Operator using Helm

Helm is the preferred method for installing the operator. The operator is available in the Debezium Helm Chart Repository. You can install the operator to an existing namespace, or to a new namespace.

Enter the following command to create a dedicated namespace:
```
kubectl create namespace debezium
```

Add the Debezium charts repository and install the operator by entering the following command:

helm repo add debezium https://charts.debezium.io
helm install my-debezium-operator debezium/debezium-operator --version 3.2.0-final -n debezium

You can find the available versions of the operator in the Debezium Helm Chart Repository.

Tracking the progress of the operator deployment

You can run commands to monitor the progress of the deployment and view the operator log.

Procedure

To monitor the progress of the deployment, run the following command:
```
kubectl get pod -n debezium --watch
```
To monitor the operator log, enter the following command:
```
kubectl logs deployment/debezium-operator -n debezium -f
```

After the operator is running, it watches for new custom resources and creates a Debezium Server instance that is based on those custom resources.

DebeziumServer custom resource

You specify the configuration for a Debezium Server instance by declaring properties in a DebeziumServer custom resource.

Example 1. Basic structure of the Debezium Server custom resource

apiVersion: debezium.io/v1alpha1
kind: DebeziumServer
metadata:
  name: my-debezium-server
spec:
  image: String
  version: String
  runtime:
    environment:
      vars:
        - name: String
          value: String
      from:  # EnvFromSource array
        - sourceName: String
          # add other EnvFromSource fields as needed
    storage:
      data:
        type: persistent | ephemeral
        claimName: String  # required if type is "persistent"
      external:  # Volume array
        - name: String
          # add other Volume fields as needed
    jmx:
      enabled: boolean
      port: int  # defaults to 1099
    metrics:
      jmxExporter:
        enabled: boolean
      openTelemetry:
        enabled: boolean
        collector:
          endpoint: String
    templates:
      container:
        resources: ResourceRequirements
        securityContext: SecurityContext
      pod:
        metadata:
          annotations:  # Map<String, String>
            key: value
          labels:  # Map<String, String>
            key: value
        imagePullSecrets:  # List
          - name: String
        affinity: Affinity
        securityContext: PodSecurityContext
  quarkus:
    config:
      # quarkus properties
      format:
        value:
          type: String
          config:
            # format properties
      key:
        type: String
        config:
          # format properties
      header:
        type: String
        config:
          # format properties
  transforms:
    - type: String
      predicate: String
      negate: Boolean
      config:
        # transformation properties
  predicates:
    name:
      type: String
      config:
        # predicate properties
  sink:
    type: String
    config:
      # sink properties
  source:
    class: String
    config:
      # source connector properties

DebeziumServer specification

DebeziumServer schema reference

Property	Type	Default	Description
`spec`	`DebeziumServerSpec`	No default value	The specification of Debezium Server
`status`	`DebeziumServerStatus`	No default value	The status of the Debezium Server instance.

Table 1. DebeziumServer properties

DebeziumServerStatus schema reference

Used in: DebeziumServer

Property	Type	Default	Description
`conditions`	`List<Condition>`	No default value	List of status conditions
`observedGeneration`	Long	0	Latest observed generation

Table 2. DebeziumServerStatus properties

Condition schema reference

Used in: DebeziumServerStatus

Property	Type	Default	Description
`status`	String	No default value	The status of the condition, either True, False or Unknown.
`message`	String	No default value	Human-readable message indicating details about the condition’s last transition.
`type`	String	No default value	Unique identifier of a condition.

Table 3. Condition properties

DebeziumServerSpec schema reference

Used in: DebeziumServer

Property	Type	Default	Description
`image`	String	No default value	Image used for Debezium Server container. This property takes precedence over version.
`version`	String	same as operator	Version of Debezium Server to be used.
`sink`	`Sink`	No default value	Sink configuration.
`source`	`Source`	No default value	Debezium source connector configuration.
`format`	`Format`	No default value	Message output format configuration.
`quarkus`	`Quarkus`	No default value	Quarkus configuration passed to the Debezium Server process.
`runtime`	`Runtime`	No default value	Specifies whether you can modify various aspects of the Debezium Server runtime.
`transforms`	`List<Transformation>`	No default value	Single Message Transformations employed by this instance of Debezium Server.
`predicates`	`Map<String, Predicate>`	No default value	Predicates employed by this instance of Debezium Server.

Table 4. DebeziumServerSpec properties

Sink schema reference

Used in: DebeziumServerSpec

Property	Type	Default	Description
`type`	String	No default value	Sink type recognized by this Debezium Server instance.
`config`	Map	No default value	Sink configuration properties.

Table 5. Sink properties

Source schema reference

Used in: DebeziumServerSpec

Property	Type	Default	Description
`sourceClass`	String	No default value	Fully qualified name of source connector Java class.
`offset`	`Offset`	No default value	Offset store configuration
`schemaHistory`	`SchemaHistory`	No default value	Schema history store configuration
`config`	Map	No default value	Source connector configuration properties.

Table 6. Source properties

Format schema reference

Used in: DebeziumServerSpec

Property	Type	Default	Description
`key`	`FormatType`	No default value	Message key format configuration.
`value`	`FormatType`	No default value	Message value format configuration.
`header`	`FormatType`	No default value	Message header format configuration.

Table 7. Format properties

FormatType schema reference

Used in: Format

Property	Type	Default	Description
`type`	String	json	Format type recognized by Debezium Server.
`config`	Map	No default value	Format configuration properties.

Table 8. FormatType properties

Quarkus schema reference

Used in: DebeziumServerSpec

Property	Type	Default	Description
`config`	Map	No default value	Quarkus configuration properties.

Table 9. Quarkus properties

Runtime schema reference

Used in: DebeziumServerSpec

Property	Type	Default	Description
`api`	`RuntimeApi`	No default value	API configuration
`storage`	`RuntimeStorage`	No default value	Storage configuration
`environment`	`RuntimeEnvironment`	No default value	Additional environment variables used by this Debezium Server instance.
`jmx`	`JmxConfig`	No default value	JMX configuration.
`templates`	`Templates`	No default value	Debezium Server resource templates.
`serviceAccount`	String	No default value	An existing service account used to run the Debezium Server pod
`metrics`	`Metrics`	No default value	Metrics configuration

Table 10. Runtime properties

RuntimeApi schema reference

Used in: Runtime

Property	Type	Default	Description
`enabled`	boolean	No default value	Whether the API should be enabled for this instance of Debezium Server
`port`	int	8080	Port number used by the k8s service exposing the API

Table 11. RuntimeApi properties

RuntimeStorage schema reference

Used in: Runtime

Property	Type	Default	Description
`data`	`DataStorage`	No default value	File storage configuration used by this instance of Debezium Server.
`external`	`List<Volume>`	No default value	Additional volumes mounted to /debezium/external

Table 12. RuntimeStorage properties

RuntimeEnvironment schema reference

Used in: Runtime

Property	Type	Default	Description
`vars`	`List<ContainerEnvVar>`	No default value	Environment variables applied to the container.
`from`	`List<EnvFromSource>`	No default value	Additional environment variables set from ConfigMaps or Secrets in containers.

Table 13. RuntimeEnvironment properties

JmxConfig schema reference

Used in: Runtime

Property	Type	Default	Description
`enabled`	boolean	false	Whether JMX should be enabled for this Debezium Server instance.
`port`	int	1099	JMX port.
`authentication`	`JmxAuthentication`	No default value	JMX authentication config.

Table 14. JmxConfig properties

JmxAuthentication schema reference

Used in: JmxConfig

Property	Type	Default	Description
`enabled`	boolean	false	Whether JMX authentication should be enabled for this Debezium Server instance.
`secret`	String	No default value	Secret providing credential files
`accessFile`	String	jmxremote.access	JMX access file name and secret key
`passwordFile`	String	jmxremote.password	JMX password file name and secret key

Table 15. JmxAuthentication properties

Templates schema reference

Used in: Runtime

Property	Type	Default	Description
`container`	`ContainerTemplate`	No default value	Container template
`pod`	`PodTemplate`	No default value	Pod template.
`volumeClaim`	`PersistentVolumeClaimSpec`	No default value	PVC template for data volume if no explicit claim is specified.

Table 16. Templates properties

Metrics schema reference

Used in: Runtime

Property	Type	Default	Description
`jmxExporter`	`JmxExporter`	No default value	Prometheus JMX exporter configuration
`openTelemetry`	`OpenTelemetry`	No default value	OpenTelemetry configuration

Table 17. Metrics properties

JmxExporter schema reference

Used in: Metrics

Property	Type	Default	Description
`enabled`	boolean	No default value	Enables JMX Prometheus exporter
`configFrom`	ConfigMapKeySelector	No default value	Config map key reference which value will be used as configuration file

Table 18. JmxExporter properties

OpenTelemetry schema reference

Used in: Metrics

When OpenTelemetry is enabled, the operator automatically configures the Debezium Server container with the following environment variables:

OTEL_ENABLED — Activates the OpenTelemetry Java Agent bundled with Debezium Server.
OTEL_SDK_DISABLED — Set to false to ensure the OpenTelemetry SDK is active.
OTEL_EXPORTER_OTLP_ENDPOINT — The OTLP collector endpoint (defaults to http://localhost:4318).
OTEL_METRICS_EXPORTER — The metrics exporter type (defaults to otlp).
OTEL_SERVICE_NAME — Set to the name of the DebeziumServer resource.
OTEL_RESOURCE_ATTRIBUTES — Includes service.name and debezium.connector.type attributes.
OTEL_JMX_CONFIG — Points to the built-in JMX-to-OpenTelemetry metrics mapping configuration.

You can override any of these environment variables by specifying them in spec.runtime.environment.vars. User-provided environment variables take precedence over operator-managed defaults.

Property	Type	Default	Description
`enabled`	boolean	false	Enables OpenTelemetry
`collector`	`OtelCollector`	No default value	OpenTelemetry collector configuration

Table 19. OpenTelemetry properties

OtelCollector schema reference

Used in: OpenTelemetry

Property	Type	Default	Description
`endpoint`	String	http://localhost:4318	OTLP collector endpoint

Table 20. OpenTelemetry collector properties

Transformation schema reference

Used in: DebeziumServerSpec

Property	Type	Default	Description
`type`	String	No default value	Fully qualified name of Java class implementing the transformation.
`config`	Map	No default value	Transformation specific configuration properties.
`predicate`	String	No default value	The name of the predicate to be applied to this transformation.
`negate`	boolean	false	Determines if the result of the applied predicate will be negated.

Table 21. Transformation properties

Predicate schema reference

Used in: DebeziumServerSpec

Property	Type	Default	Description
`type`	String	No default value	Fully qualified name of Java class implementing the predicate.
`config`	Map	No default value	Predicate configuration properties.

Table 22. Predicate properties

ConfigMapOffsetStore schema reference

Used in: Offset

Property	Type	Default	Description
`name`	String	No default value	Name of the offset config map
`config`	Map	No default value	Additional store configuration properties.

Table 23. ConfigMapOffsetStore properties

ContainerEnvVar schema reference

Used in: RuntimeEnvironment

Property	Type	Default	Description
`name`	String	No default value	The environment variable name.
`value`	String	No default value	The environment variable value.

Table 24. ContainerEnvVar properties

ContainerTemplate schema reference

Used in: Templates

Property	Type	Default	Description
`resources`	`ResourceRequirements`	No default value	CPU and memory resource requirements.
`securityContext`	`SecurityContext`	No default value	Container security context.
`probes`	`Probes`	No default value	Container probes configuration.
`imagePullPolicy`	Always,IfNotPresent,Never	No default value	Image pull policy for the container.

Table 25. ContainerTemplate properties

CustomStore schema reference

Used in: Offset, SchemaHistory

Property	Type	Default	Description
`type`	String	No default value	Fully qualified name of Java class implementing the store.
`config`	Map	No default value	Store configuration properties.

Table 26. CustomStore properties

DataStorage schema reference

Used in: RuntimeStorage

Property	Type	Default	Description
`type`	ephemeral,persistent	ephemeral	Storage type.
`claimName`	String	No default value	Name of persistent volume claim for persistent storage.

Table 27. DataStorage properties

FileOffsetStore schema reference

Used in: Offset

Property	Type	Default	Description
`fileName`	String	No default value	Name of the offset file (relative to data root)
`config`	Map	No default value	Additional store configuration properties.

Table 28. FileOffsetStore properties

FileSchemaHistoryStore schema reference

Used in: SchemaHistory

Property	Type	Default	Description
`fileName`	String	No default value	Name of the offset file (relative to data root)
`config`	Map	No default value	Additional store configuration properties.

Table 29. FileSchemaHistoryStore properties

InMemoryOffsetStore schema reference

Used in: Offset

Property	Type	Default	Description
`config`	Map	No default value	Additional store configuration properties.

Table 30. InMemoryOffsetStore properties

InMemorySchemaHistoryStore schema reference

Used in: SchemaHistory

Property	Type	Default	Description
`config`	Map	No default value	Additional store configuration properties.

Table 31. InMemorySchemaHistoryStore properties

JdbcOffsetStore schema reference

Used in: Offset

Property	Type	Default	Description
`table`	`JdbcOffsetTableConfig`	No default value	The configuration of the offset table
`url`	String	No default value	JDBC connection URL
`user`	String	No default value	Username used to connect to the storage database
`password`	String	No default value	Password used to connect to the storage database
`retryDelay`	long	No default value	Retry delay on connection failure (in milliseconds)
`maxRetries`	int	No default value	Maximum number of retries on connection failure
`config`	Map	No default value	Additional store configuration properties.

Table 32. JdbcOffsetStore properties

JdbcOffsetTableConfig schema reference

Used in: JdbcOffsetStore

Property	Type	Default	Description
`name`	String	No default value	The name of the offset table
`ddl`	String	No default value	DDL statement to create the offset table
`select`	String	No default value	Statement used to select from the offset table
`insert`	String	No default value	Statement used to insert into the offset table
`delete`	String	No default value	Statement used to update the offset table

Table 33. JdbcOffsetTableConfig properties

JdbcSchemaHistoryStore schema reference

Used in: SchemaHistory

Property	Type	Default	Description
`table`	`JdbcSchemaHistoryTableConfig`	No default value	The configuration of the offset table
`url`	String	No default value	JDBC connection URL
`user`	String	No default value	Username used to connect to the storage database
`password`	String	No default value	Password used to connect to the storage database
`retryDelay`	long	No default value	Retry delay on connection failure (in milliseconds)
`maxRetries`	int	No default value	Maximum number of retries on connection failure
`config`	Map	No default value	Additional store configuration properties.

Table 34. JdbcSchemaHistoryStore properties

JdbcSchemaHistoryTableConfig schema reference

Used in: JdbcSchemaHistoryStore

Property	Type	Default	Description
`name`	String	No default value	The name of the offset table
`ddl`	String	No default value	DDL statement to create the schema history table
`select`	String	No default value	Statement used to select from the schema history table
`insert`	String	No default value	Statement used to insert into the schema history table
`dataExistsSelect`	String	No default value	Statement used to check existence of some data in the schema history table

Table 35. JdbcSchemaHistoryTableConfig properties

KafkaOffsetStore schema reference

Used in: Offset

Property	Type	Default	Description
`props`	Map	No default value	Additional Kafka client properties.
`bootstrapServers`	String	No default value	A list of host/port pairs that the connector uses for establishing an initial connection to the Kafka cluster
`topic`	String	No default value	The name of the Kafka topic where offsets are to be stored
`partitions`	int	No default value	The number of partitions used when creating the offset storage topic
`replicationFactor`	int	No default value	Replication factor used when creating the offset storage topic
`config`	Map	No default value	Additional store configuration properties.

Table 36. KafkaOffsetStore properties

KafkaSchemaHistoryStore schema reference

Used in: SchemaHistory

Property	Type	Default	Description
`bootstrapServers`	String	No default value	A list of host/port pairs that the connector uses for establishing an initial connection to the Kafka cluster
`topic`	String	No default value	The name of the Kafka topic where offsets are to be stored
`partitions`	int	No default value	The number of partitions used when creating the offset storage topic
`replicationFactor`	int	No default value	Replication factor used when creating the offset storage topic
`config`	Map	No default value	Additional store configuration properties.

Table 37. KafkaSchemaHistoryStore properties

MetadataTemplate schema reference

Used in: PodTemplate

Property	Type	Default	Description
`labels`	Map<String, String>	No default value	Labels added to the Kubernetes resource
`annotations`	Map<String, String>	No default value	Annotations added to the Kubernetes resource

Table 38. MetadataTemplate properties

Offset schema reference

Used in: Source

Property	Type	Default	Description
`file`	`FileOffsetStore`	No default value	File backed offset store configuration
`memory`	`InMemoryOffsetStore`	No default value	Memory backed offset store configuration
`redis`	`RedisOffsetStore`	No default value	Redis backed offset store configuration
`kafka`	`KafkaOffsetStore`	No default value	Kafka backing store configuration
`jdbc`	`JdbcOffsetStore`	No default value	JDBC backing store configuration
`configMap`	`ConfigMapOffsetStore`	No default value	Config map backed offset store configuration
`store`	`CustomStore`	No default value	Arbitrary offset store configuration
`flushMs`	long	60000	Interval at which to try commiting offsets

Table 39. Offset properties

PodTemplate schema reference

Used in: Templates

Property	Type	Default	Description
`metadata`	`MetadataTemplate`	No default value	Metadata applied to the resource.
`imagePullSecrets`	`List<LocalObjectReference>`	No default value	List of local references to secrets used for pulling any of the images used by this Pod.
`affinity`	`Affinity`	No default value	Pod affinity rules
`securityContext`	`PodSecurityContext`	No default value	Pod-level security attributes and container settings

Table 40. PodTemplate properties

Probe schema reference

Used in: Probes

Property	Type	Default	Description
`initialDelaySeconds`	int	5	Number of seconds after the container has started before probes are initiated.
`periodSeconds`	int	10	How often (in seconds) to perform the probe.
`timeoutSeconds`	int	10	Number of seconds after which the probe times out.
`failureThreshold`	int	3	Number of failures in a row before the overall check has failed.

Table 41. Probe properties

Probes schema reference

Used in: ContainerTemplate

Property	Type	Default	Description
`readiness`	`Probe`	No default value	Readiness probe configuration applied to the container.
`liveness`	`Probe`	No default value	Liveness probe configuration applied to the container.

Table 42. Probes properties

RedisOffsetStore schema reference

Used in: Offset

Property	Type	Default	Description
`address`	String	No default value	Redis host:port used to connect
`user`	String	No default value	Redis username
`password`	String	No default value	Redis password
`sslEnabled`	boolean	false	Redis username
`key`	String	No default value	Redis hash key
`wait`	`RedisStoreWaitConfig`	No default value	Configures verification of replica writes
`config`	Map	No default value	Additional store configuration properties.

Table 43. RedisOffsetStore properties

RedisSchemaHistoryStore schema reference

Used in: SchemaHistory

Property	Type	Default	Description
`address`	String	No default value	Redis host:port used to connect
`user`	String	No default value	Redis username
`password`	String	No default value	Redis password
`sslEnabled`	boolean	false	Redis username
`key`	String	No default value	Redis hash key
`wait`	`RedisStoreWaitConfig`	No default value	Configures verification of replica writes
`config`	Map	No default value	Additional store configuration properties.

Table 44. RedisSchemaHistoryStore properties

RedisStoreWaitConfig schema reference

Used in: RedisOffsetStore, RedisSchemaHistoryStore

Property	Type	Default	Description
`enabled`	boolean	false	In case of Redis with replica, this allows to verify that the data has been written to replica
`timeoutMs`	long	1000	Timeout in ms when waiting for replica
`retry`	boolean	false	Enables retry on wait for replica
`retryDelayMs`	long	1000	Delay of retry on wait

Table 45. RedisStoreWaitConfig properties

SchemaHistory schema reference

Used in: Source

Property	Type	Default	Description
`file`	`FileSchemaHistoryStore`	No default value	File backed schema history store configuration
`memory`	`InMemorySchemaHistoryStore`	No default value	Memory backed schema history store configuration
`redis`	`RedisSchemaHistoryStore`	No default value	Redis backed schema history store configuration
`kafka`	`KafkaSchemaHistoryStore`	No default value	Kafka backed schema history store configuration
`jdbc`	`JdbcSchemaHistoryStore`	No default value	JDBC backed schema history store configuration
`store`	`CustomStore`	No default value	Arbitrary schema history store configuration
`config`	Map	No default value	Additional common schema history store configuration properties.

Table 46. SchemaHistory properties

Complete Debezium Server configuration example

The following example shows a complete Debezium Server custom resource that incorporates properties from the preceding tables.

Example 2. Complete Debezium server custom resource

apiVersion: debezium.io/v1alpha1
kind: DebeziumServer
metadata:
  name: my-debezium-server
spec:
  version: "3.2.0"
  source:
    class: io.debezium.connector.mysql.MySqlConnector
    config:
      database.hostname: mysql-server
      database.port: 3306
      database.user: debezium
      database.password: secret
      database.server.id: 184054
      topic.prefix: mysql
      database.include.list: inventory
    offset:
      type: kafka
      config:
        bootstrap.servers: kafka:9092
        topic: debezium-offsets
    schemaHistory:
      type: kafka
      config:
        bootstrap.servers: kafka:9092
        topic: debezium-schema-history
  sink:
    type: kafka
    config:
      bootstrap.servers: kafka:9092
  format:
    key:
      type: json
    value:
      type: json
      config:
        schemas.enable: false
  runtime:
    jmx:
      enabled: true
      port: 1099
    storage:
      data:
        type: persistent
        claimName: debezium-data-pvc
    metrics:
      jmxExporter:
        enabled: true
      openTelemetry:
        enabled: true
        collector:
          endpoint: http://otel-collector:4318
  transforms:
    - type: io.debezium.transforms.UnwrapFromEnvelope
      config:
        drop.tombstones: false
      predicate: inventory-filter
      negate: false
  predicates:
    inventory-filter:
      type: org.apache.kafka.connect.transforms.predicates.TopicNameMatches
      config:
        pattern: "mysql.inventory.*"
  quarkus:
    config:
      quarkus.log.level: INFO
      quarkus.http.port: 8080

Debezium Operator

Debezium Operator

Overview

Deploy Debezium Operator using Helm

Tracking the progress of the operator deployment

DebeziumServer custom resource

DebeziumServer specification

DebeziumServer schema reference

DebeziumServerStatus schema reference

Condition schema reference

DebeziumServerSpec schema reference

Sink schema reference

Source schema reference

Format schema reference

FormatType schema reference

Quarkus schema reference

Runtime schema reference

RuntimeApi schema reference

RuntimeStorage schema reference

RuntimeEnvironment schema reference

JmxConfig schema reference

JmxAuthentication schema reference

Templates schema reference

Metrics schema reference

JmxExporter schema reference

OpenTelemetry schema reference

OtelCollector schema reference

Transformation schema reference

Predicate schema reference

ConfigMapOffsetStore schema reference

ContainerEnvVar schema reference

ContainerTemplate schema reference

CustomStore schema reference

DataStorage schema reference

FileOffsetStore schema reference

FileSchemaHistoryStore schema reference

InMemoryOffsetStore schema reference

InMemorySchemaHistoryStore schema reference

JdbcOffsetStore schema reference

JdbcOffsetTableConfig schema reference

JdbcSchemaHistoryStore schema reference

JdbcSchemaHistoryTableConfig schema reference

KafkaOffsetStore schema reference

KafkaSchemaHistoryStore schema reference

MetadataTemplate schema reference

Offset schema reference

PodTemplate schema reference

Probe schema reference

Probes schema reference

RedisOffsetStore schema reference

RedisSchemaHistoryStore schema reference

RedisStoreWaitConfig schema reference

SchemaHistory schema reference

Complete Debezium Server configuration example

评论