Debezium Management Platform

Debezium Management Platform aims to simplify the deployment of Debezium to various environments in a highly opinionated manner. To achieve this goal, the platform uses a data-centric view of Debezium components.

Implementing the platform represents a natural evolution from Debezium Server. Past releases provided the Debezium operator to simplify operation in Kubernetes environments. With the introduction of the platform, Debezium now provides a high-level abstraction to deploy your data pipelines in different environments while leveraging Debezium Server.

Basic concepts

You can use the Debezium Management Platform to configure and manage the following components in your environment:

Source

Specifies the database from which you want Debezium to read data.

Destination

Specifies the sink destination where you want Debezium to send data.

Connection

Specifies the connection details that Debezium needs to communicate with a data source or destination.

Transform

Specifies how Debezium transforms a data event as it flows through the pipeline.

Pipeline

Specifies how data flows from a source to a destination while being transformed along the way. Each pipeline consists of a source, one or more transforms, and a destination.

After you define a pipeline, it is deployed based on how you configure the platform.

Each pipeline is mapped to a Debezium Server instance. For the Kubernetes environment (currently, the only supported environment) the server instance corresponds to a DebeziumServer custom resource.

Debezium Platform

Architecture

The platform is composed of the following components:

Conductor

The back-end component that provides a set of APIs to orchestrate and control Debezium deployments.

Stage

The front-end component that provides a user interface to interact with the Conductor.

The conductor component itself is composed of the following subcomponents:

API Server

The main entry point. It provides a set of APIs to interact with the platform.

Watcher

The component that is responsible for the actual communication with the deployment environment (for example, the Debezium Operator in a Kubernetes cluster).

Installation

Prerequisites

• Helm
• Kubernetes cluster with an ingress controller

Installation is provided through a Helm chart.

Procedure

1. Enter the following command to add the Debezium charts repository:

   helm repo add debezium https://charts.debezium.io

2. Enter one of the following commands to install the latest stable version of the platform that you want:

   helm install debezium-platform debezium/debezium-platform --set database.enabled=true --set domain.name=platform.debezium.io

   Or, to use an OCI artifact to install the platform, enter the following command:

   helm install debezium-platform --set database.enabled=true --set domain.name=platform.debezium.io oci://quay.io/debezium-charts/debezium-platform

   The domain.name is the only required property; it is used as host in the Ingress definition.

   In the preceding examples, the database.enabled property is used. This property helps to simplify deployment in testing environments by automatically deploying the PostgreSQL database that is required by the conductor service. When deploying in a production environment, do not enable automatic deployment of the PostgreSQL database. Instead, specify an existing database instance, by setting the database.name, database.host, and other properties required to connect to the database. See the following table for more information.

The following table lists all the chart’s properties:

Name Description Default

Domain

domain.name

Domain used as the ingress host.

""

domain.url

Domain used as the ingress host (DEPRECATED). Use domain.name instead.

""

Stage

stage.image

Image that Helm uses to deploy the stage (UI) pod.

quay.io/debezium/platform-stage:<release_tag>

stage.imagePullPolicy

Image pull policy for the stage container. If empty, defaults to IfNotPresent.

IfNotPresent

Conductor

conductor.image

Image that Helm uses to deploy the conductor pod.

quay.io/debezium/platform-conductor:<release_tag>

conductor.imagePullPolicy

Image pull policy for the conductor container. If empty, defaults to IfNotPresent.

IfNotPresent

conductor.offset.existingConfigMap

Name of the ConfigMap that stores conductor offsets. If no value is specified, Helm creates a ConfigMap automatically.

""

conductor.descriptors.official.enabled

Enables official Debezium descriptors, which are downloaded via ORAS at startup.

true

conductor.descriptors.official.registry

OCI registry hosting the descriptor artifact.

quay.io

conductor.descriptors.official.image

Image name for the descriptor OCI artifact.

debezium/debezium-descriptors

conductor.descriptors.official.tag

Image tag for the descriptor OCI artifact.

nightly

conductor.descriptors.official.mountPath

Path where descriptors are downloaded inside the container.

/opt/descriptors

conductor.extraVolumes

Extra volumes to add to the conductor deployment.

[]

conductor.extraVolumeMounts

Extra volume mounts to add to the conductor container.

[]

Server

server.image

Image for Debezium Server instances created by pipelines. If empty, the operator’s ServerImageProvider determines the image.

""

Ingress

ingress.enabled

Enables the ingress resource for conductor and stage.

true

ingress.className

Optional ingress class name.

""

ingress.annotations

Extra ingress annotations.

{}

ingress.tls.enabled

Enables the TLS section on the ingress.

false

ingress.tls.secretName

Secret name used when TLS is enabled.

""

Database

database.enabled

Enables Helm to install PostgreSQL.

false

database.name

Name of an existing database where you want the platform to store data.

postgres

database.host

Host of the database that you want the platform to use.

postgres

database.auth.existingSecret

Name of the secret that stores the username and password that the platform uses to authenticate with the database. If no value is specified, Helm automatically creates a secret based on the credentials that you provide in the database.auth.username and database.auth.password properties.

If you provide a value for this property, do not set database.auth.username or database.auth.password.

""

database.auth.username

Username through which the platform connects to the database.

user

database.auth.password

Password for the user specified by database.auth.username.

password

Offset storage

offset.reusePlatformDatabase

Specifies whether pipelines use the configured platform database to store offsets. To configure pipelines to use a different, dedicated database to store offsets, set the value to false.

true

offset.database.name

Name of the database that the platform uses to store offsets.

postgres

offset.database.host

Host for the database where the platform stores offsets.

postgres

offset.database.port

Port through which the platform connects to the database where it stores offsets.

5432

offset.database.auth.existingSecret

Name of the secret that stores the username and password that the platform uses to authenticate with the database that stores offsets. If you do not specify a value, the platform uses the values of the offset.database.auth.username and offset.database.auth.password properties to authenticate with the database.

If you provide the name of a secret, do not set the offset.database.auth.username and offset.database.auth.password properties.

""

offset.database.auth.username

Username through which the platform connects to the offsets database.

user

offset.database.auth.password

Password for the offsets database user specified by offset.database.auth.username.

password

Schema history storage

schemaHistory.reusePlatformDatabase

Specifies whether pipelines use the configured platform database to store the schema history. To configure pipelines to use a different, dedicated database to store the schema history, set the value to false.

true

schemaHistory.database.name

Name of the dedicated database where the platform stores the schema history.

postgres

schemaHistory.database.host

Host for the dedicated database where the platform stores the schema history.

postgres

schemaHistory.database.port

Port through which the platform connects to the dedicated database where it stores the schema history.

5432

schemaHistory.database.auth.existingSecret

Name of the secret that stores the username and password that the platform uses to authenticate with the database that stores the schema history. If you do not specify a value, the platform uses the values of the schemaHistory.database.auth.username and schemaHistory.database.auth.password properties to authenticate with the database.

If you provide the name of a secret, do not set the schemaHistory.database.auth.username and schemaHistory.database.auth.password properties.

""

schemaHistory.database.auth.username

Username through which the platform connects to the schema history database.

user

schemaHistory.database.auth.password

Password for the schema history database user specified by schemaHistory.database.auth.username property.

password

Debezium Operator

debezium-operator.enabled

Enables the installation of the Debezium Operator by the chart.

true

Pipeline

pipeline.labels

Map of labels to apply to DebeziumServer custom resources created by pipelines. These labels are merged with the internal debezium.io/conductor-id label.

{}

Monitoring

monitoring.otel.enabled

Enables the OpenTelemetry monitoring infrastructure. Requires the OpenTelemetry Operator to be installed. For more information, see Enabling monitoring.

false

monitoring.otel.collector.image

OpenTelemetry Collector image. Must include the Prometheus exporter. If empty, the operator’s default image is used.

""

monitoring.otel.collector.replicas

Number of OpenTelemetry Collector replicas.

1

monitoring.otel.collector.receivers.grpc.port

Port for the OTLP gRPC receiver.

4317

monitoring.otel.collector.receivers.http.port

Port for the OTLP HTTP receiver.

4318

monitoring.otel.collector.processors.batch.timeout

Batch processor flush timeout.

5s

monitoring.otel.collector.processors.batch.sendBatchSize

Maximum number of metrics per batch.

512

monitoring.otel.collector.exporters.prometheus.port

Port on which the Prometheus exporter listens.

8889

monitoring.otel.collector.exporters.prometheus.resourceToTelemetryConversion

Converts OpenTelemetry resource attributes to Prometheus labels.

true

monitoring.otel.collector.exporters.prometheus.constLabels

Static labels added to all exported metrics.

{platform: debezium}

monitoring.panels.additionalPanelsPath

Path to a YAML file that contains additional monitoring panels. Panels are merged with built-in defaults; matching IDs override built-in panels. For more information, see Custom monitoring panels.

""

monitoring.prometheus.url

URL of the Prometheus instance used by the Conductor to query metrics. Required when monitoring.otel.enabled is true.

""

monitoring.prometheus.serviceMonitor.enabled

Creates a ServiceMonitor for automatic Prometheus scraping. Requires the Prometheus Operator to be installed.

true

monitoring.prometheus.serviceMonitor.scrapeInterval

Prometheus scrape interval.

15s

monitoring.prometheus.serviceMonitor.labels

Labels for Prometheus Operator ServiceMonitor discovery. Must match the serviceMonitorSelector labels configured in your Prometheus instance.

{prometheus: kube-prometheus}

Other

env

List of environment variables to pass to the conductor.

[]

Using the platform

You can use the platform UI to perform a number of different tasks.

Defining connections

Use the Connection section of the UI to define the connections to the database that hosts your source data and to your data sink. You can configure connections to any database that Debezium supports as either a data source or destination sink. The connections that you create can be shared by multiple sources and destinations.

Creating a connection

You can create a connection for any source or destination that is available in the connection catalog. The connection catalog lists the source databases that Debezium can read from, and the sink destinations that Debezium Server can write to.

Procedure

1. From the platform UI, open the Connection menu.
2. Click Add connection to open the Connection catalog page.
3. Click Source or Destination to filter the list by connection type, and then select the required connection card.
4. In the configuration form, specify the connection name and predefined properties, and specify any custom properties in the Additional properties section.
5. Click Verify to validate the configuration and ensure that Debezium can establish a connections with the database source or sink.
6. After you verify the connection, click Create connection .

Deleting a connection

Prerequisites

• The connection that you want to delete is not in use in any source or destination.

Procedure

• From the platform UI, open the Connection menu, click the Action menu for the connection that you want to delete, and then click Delete.

An error results if you attempt to delete a connection that is in use. If the operation returns an error, verify that the connection is no longer used in any sources or destinations, and then repeat the delete operation.

Editing a connection

To edit a connection, from the platform UI, open the Connection menu, click the Action menu for the connection that you want to edit, and then click Edit.

Defining sources

Use the Source section of the UI to specify the database that hosts your data. You can configure any database that Debezium supports as a source. The source that you create can be shared among multiple pipelines. Changes to a source are reflected in every pipeline that uses it.

Creating a source

You can use either of the following editors to configure a source:

Form Editor

Enables you to specify the name, description, and connection of the source, along with a list of properties. For a connection, you can select the predefined source connection, or click Create connection to create a new connection. For more information, see Creating a connection. For a complete list of the properties that are available for a connector, see the connector documentation.

Smart Editor

Provides a way to define the source configuration in JSON format. You can also use the editor to reformat source configurations that are designed for Debezium connectors in Kafka Connect or Debezium Server environment for use in the platform. The editor automatically applies the JSON formatting that the platform requires to the specified source configuration.

You can create source configurations based on existing Debezium configurations, such as the configuration for a Debezium connector for Kafka Connect, or a Debezium Server configuration.

From the toolbar on the Source catalog page, select Create using smart editor .

Create, edit and remove a source

Configuring a source with the Smart Editor

You can use the Smart Editor to specify the JSON that defines the source configuration. You can enter and edit JSON directly in the editor, or paste JSON from an external source into the editor. The JSON that you use to configure a data source in the Debezium management platform is nearly identical to the JSON in the config section that defines the configuration for a Debezium connector on Kafka Connect or in the Debezium Server.

To reuse an existing Debezium connector configuration, you can upload a file that contains the configuration, or you can copy and paste the configuration into the Smart Editor. Stage UI intelligently recognizes the format of the configuration that you provide, and prompts you to reformat it for use by the platform.

For example, consider the following JSON for specifying the configuration of a Debezium MySQL connector on Kafka Connect:

{
  "name": "inventory-connector",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",
    "database.hostname": "mysql",
    "database.port": "3306",
    "database.user": "debezium",
    "database.password": "dbz",
    "database.server.id": "184054",
    "topic.prefix": "dbserver1",
    "database.include.list": "inventory"
  }
}

When you add this JSON to the Smart Editor, it detects that it is formatted for Kafka Connect and displays an alert.

The Smart Editor toolbar then displays an Auto Format option, as shown in the preceding figure. The Autoformat option automatically converts the provided JSON into the required Debezium Platform JSON format.

The following image shows the JSON that results after you use the Autoformat option to convert the configuration for a MySQL data source. Optionally, you can edit the JSON to update the value of the name field or populate the` description` field.

Th following example shows the platform configuration JSON after updates to the name and description fields.

{
    "name": "my-source",
    "description": "This is my first source",
    "type": "io.debezium.connector.mysql.MySqlConnector",
    "schema": "schema123",
    "vaults": [],
    "config": {
        "database.hostname": "mysql",
        "database.port": "3306",
        "database.user": "debezium",
        "database.password": "dbz",
        "database.server.id": "184054",
        "topic.prefix": "dbserver1",
        "database.include.list": "inventory"
    }
}

Similarly, you can open a Debezium Server configuration in the Smart Editor to automatically extract the source configuration and reformat it for use in the Debezium platform. For more information, see Configuring a destination with the Smart Editor.

Deleting a source

Prerequisites

• The source that you want to delete is not in use in any pipeline.

Procedure

• From the platform UI, open the Source menu, click the Action menu for the source that you want to delete, and then click Delete.

An error results if you attempt to delete a source that is in use. If the operation returns an error, verify that the source is no longer used in any pipeline, and then repeat the delete operation.

Editing a source

To edit a source, from the platform UI, open to the Source menu, click the Action menu of the source that you want to edit, and then click Edit.

Editing a source affects all pipelines that use it. Before you submit a change, a message notifies you that pipelines that use the source will restart.

Enabling signaling

Prerequisites

• You have sufficient access privileges to create a table on the source database.

The Debezium signaling mechanism provides a way to modify the behavior of a connector, or to trigger a one-time action. In the Debezium management platform, signaling actions are performed via the source connector’s SourceSignalChannel. For more information about signaling, see Enabling the source signaling channel.

To enable the signaling for a data pipeline, you must set up signaling in the source connector configuration.

The signaling data collection, which you create on the source database, is designated exclusively for communicating with Debezium. The signaling data collection must be unique to each connector instance.

1. In the dialog box, provide the Signaling collection name and run the provided DDL query to create a signaling data collection in your source database.
2. Click Verify to check whether the signaling data collection is set up correctly.
3. Click Done to save the signaling collection.

Defining destinations

Use the Destination section of the UI to specify the data sink to which the platform sends source data. All Debezium Server sinks are available as destination. When you create a destination, it can be shared between different pipelines, which means that every change to a destination will be reflected in every pipeline that uses it.

Create, edit and remove a destination

Creating a destination

Use the Destination section of the UI to configure the sink destinations to which the platform sends data. You can use either of the following editors to configure a destination:

Form Editor

Enables you to specify the name, description, and connection of the destination, along with a list of properties. For a connection, you can select the predefined source connection, or click Create connection to create a new connection. For more information, see Creating a connection. For a complete list of the properties that are available for a sink connector, see the connector documentation.

Smart Editor

Enables you to define the sink configuration in JSON. Alternately, you can extract a Debezium Server sink configuration and automatically reformat it for use in the platform.

To create a sink destination from an existing Debezium Server configuration, from the Destination catalog page, click Create using smart editor .

Configuring a destination with the Smart Editor

You can use the Smart Editor to specify the JSON that defines the source configuration. You can enter and edit JSON directly in the editor, or paste JSON from an external source into the editor. With a few small differences, the JSON that you use to configure a destination in the Smart Editor is nearly identical to the configuration that you use to define a Debezium Server sink.

You can directly use a Debezium Server configuration by either uploading it, or pasting it into the Smart Editor. The Stage UI intelligently recognizes the configuration type and prompts you to reformat it for use by the platform. The editor automatically extracts the Debezium server sink configuration and converts it into the JSON format that the platform supports.

For example, consider the following properties from a Debezium Server configuration:

# ...

debezium.sink.type=pubsub
debezium.sink.pubsub.project.id=debezium-tutorial-local
debezium.sink.pubsub.address=pubsub:8085
debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
debezium.source.database.hostname=mysql
debezium.source.database.server.id=223344
debezium.source.database.port=3306
debezium.source.database.user=debezium
debezium.source.database.password=dbz
debezium.source.schema.history.internal=io.debezium.storage.file.history.FileSchemaHistory
debezium.source.schema.history.internal.file.filename=data/schema.dat
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=0
debezium.source.topic.prefix=tutorial
debezium.source.database.include.list=inventory
debezium.source.table.include.list=inventory.customers

# ..

When you add the preceding properties file to the Smart Editor, it confirms that it recognizes the file as a Debezium Server configuration.

The Smart Editor toolbar displays an Auto Format option, as shown in the preceding image. Select this option to automatically extract the sink configuration and convert the sink properties into the JSON format that the Debezium Platform supports.

The following image shows the JSON that results after the editor automatically formats the publish-subscribe sink properties that you added. Edit the JSON to update the value of the name field, and optionally populate the description field.

After you update the name and description, the following JSON results:

{
  "name": "my-sink",
  "description": "This is my first sink",
  "type": "pubsub",
  "schema": "schema123",
  "vaults": [],
  "config": {
    "debezium.sink.pubsub.project.id": "debezium-tutorial-local",
    "debezium.sink.pubsub.address": "pubsub:8085"
  }
}

Deleting a destination

Prerequisites

• The sink that you want to delete is not in use in any pipeline.

Procedure

• From the platform UI, open the Destination menu, click the Action menu of the destination you want to delete, and then click Delete.

An error results if you attempt to delete a destination that is in use. If the operation returns an error, verify that the destination is no longer used in any pipeline, and then repeat the delete operation.

Editing a destination

To edit a destination, go to the Destination menu and then click the action menu of the destination you want to edit, then click Edit.

Managing transforms

Use the Transforms section of the platform UI to manage the transformations that you want to use in your data pipeline.

Currently, the platform supports all single message transformations provided by Debezium as well as any Kafka Connect transformations.

Transformations are shared among pipelines. When you modify a transformation, the changes are reflected in all pipelines that use the transformation.

Create, edit and remove a transform

Creating a transformation

Use the Transforms section of the platform UI to specify the configure and manage single message transformations.

You can use either of the following editors to configure transformations:

Form Editor

Enables you to specify the name, type, and description of the transformation. You can also set additional configuration options that are specific to the transform type.

Optionally, if you want to apply the transformation only to records that meet specific criteria, you can specify a predicate. You can choose the predicate from a list, and set its properties.

Smart Editor

Enables you to use JSON to configure the transformation.

Using the Smart Editor to configure transformations

You can use the Smart Editor to specify the JSON that defines the transform configuration. You can enter and edit JSON directly in the editor, or paste JSON from an external source into the editor.

The format for configuring transformations in the Smart Editor differs from the Kafka Connect format that Debezium uses to configure transformations, but you can easily convert between formats.

Typically, entries in the configuration of a transformation are prefixed with transforms.<transform_name> where <transform_name is the name assigned to the transformation.

For example, in Debezium, the following configuration is used with the unwrap (ExtractNewRecordState) transformation:

# ...

transforms=unwrap
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.add.fields=op
transforms.unwrap.add.headers=db,table
predicates=onlyProducts
predicates.onlyProducts.type=org.apache.kafka.connect.transforms.predicates.TopicNameMatches
predicates.onlyProducts.pattern=inventory.inventory.products

# ..

To adapt this configuration for use in Debezium platform, convert the properties that include the prefix transforms.unwrap, except for transforms.unwrap.type, to JSON format. Apply the same process to convert predicate statements.

After you convert the Debezium configuration for the unwrap transformation, the following JSON results:

{
  "name": "Debezium marker",
  "description": "Extract Debezium payloa d",
  "type": "io.debezium.transforms.ExtractNewRecordState",
  "schema": "string",
  "vaults": [],
  "config": {
    "add.fields": "op",
    "add.headers": "db,table"
  },
  "predicate": {
    "type": "org.apache.kafka.connect.transforms.predicates.TopicNameMatches",
    "config": {
      "pattern": "inventory.inventory.products"
    },
    "negate": false
  }
}

Editing transformations

From the platform UI, open the Transform menu, click the Action menu for the transformation that you want to edit, and then click Edit.

Deleting a transformation

Prerequisites

• The transformation that you want to delete is not in use in any pipeline.

Procedure

• From the platform UI, open the Transform menu, click the Action menu of the transformation you want to delete, and then click Delete.

  An error results if you attempt to delete a transformation that is in use. If the operation returns an error, verify that the transformation is no longer used in any pipeline, and then repeat the delete operation.

Creating and managing pipelines

The pipeline section is the place where you connect the "dots". You can define where your data comes, how to eventually transform them and where they should go.

Create, edit and remove a pipeline

Creating a pipeline

1. From the platform UI, open the Pipeline menu, and then click Create your first pipeline. The Pipeline Designer opens. From the Pipeline designer, specify the components that you want to add to your data pipeline.
2. Click the + Source box to add a source, and then choose a previously created source, or create a new source.
3. (Optional) Click the + Transform box to apply one or more transformations.
4. Click the + Destination box to add a destination,and then choose a previously created destination, or create a new destination.

The sequence in which Debezium applies transformations to the source is significant and affects the final output. Verify that the sequence that you specify results in the expected output.

In the Pipeline designer* you can delete a transform, or alter the sequence of applied transforms. For more information about using the Pipeline designer, see using the Pipeline designer to remove and order transformations.

	Transformations that are configured with a predicate are marked with a predicate icon (). A tooltip shows the name of the predicate type.

After you finish designing your pipeline, click Configure Pipeline, and then specify the name, description, and logging level for the pipeline.

Creating a pipeline from a Debezium Server configuration

The Pipeline designer provides an option to create data pipeline resources (that is, source, destination, and transforms) by uploading an existing Debezium Server configuration properties file .

• In the Pipeline designer, click DBZ server config, upload, or drag and drop a Debezium server properties file, and then click Create. The designer creates the Source, Transforms, and Destination for the pipeline, based on the Debezium server configuration that you provided.

Deleting a pipeline

From the platform UI, open the Pipeline menu, click the Action menu for the pipeline that you want to delete, and then click Delete. The deletion removes only the pipeline: the source, destination, and any transformations are not deleted.

Editing a pipeline

1. From the platform UI, open the Pipeline menu, click the Action menu for the pipeline that you want to edit, and then click Edit.
2. In the Pipeline designer, modify transformations as needed. For more information about using the pipeline designer, see Using the Pipeline designer to remove and order transformations.
3. Click Save and next to edit the name, description, and log level properties of the pipeline.

Using the Pipeline designer to remove and order transformations

From the Pipeline designer, you can delete transformations, or rearrange the order in which they run.

Deleting a transformation

1. From the Pipeline designer, click the pencil icon () in the Transform box.
2. From the Transform list, click the trash icon next to the name of the transform.

Rearranging transformations

If you configure a connector to use multiple transformations, you can use the Transform list in the Pipeline designer to specify the order in which they are applied. The first transformation in the list processes the message first.

1. From the Pipeline designer, click the pencil icon () in the Transform box.
2. From the Transform list, drag transformations into the order in which you want to apply them, and then click Apply.

Monitoring the pipeline

To assist you in monitoring activity in the pipeline, the platform provides you with quick access to pipeline logs.

Procedure

1. From the platform UI, click Pipeline, click the name of the pipeline you want to monitor, and then click Pipeline logs.

1. Download the logs, toggle to full screen view, search the log for a relevant section, or click Grep notifications to search for the [Notification Service].

Pipeline actions

To modify data pipeline components, you can use Debezium signaling to trigger specific actions.

Prerequisites

• Signaling is enabled for the source connector. For more information see Enabling signaling.

Procedure

1. From the Platform UI, open the pipeline overview page, and click the Pipeline actions tab.

1. Select an action from the dropdown list. A form appears that shows fields that are relevant to the selected action.

2. Edit the fields as needed.

   	The Action Id field is prefilled with a UUID string. You can update the default value, but always ensure that the value is unique for each action.

3. After you finish editing the form, click Submit to trigger an action.

Pipeline monitoring

The Debezium Management Platform provides built-in monitoring capabilities that enable you to observe pipeline health and performance directly from the Stage UI. By using the integrated monitoring dashboard, you can track key metrics such as event throughput, replication lag, queue utilization, and snapshot progress without requiring external monitoring tools or separate authentication.

The monitoring infrastructure is based on industry-standard open-source components: OpenTelemetry for metrics collection and Prometheus for metrics storage.

Monitoring architecture

The monitoring system consists of the following components working together to collect, store, and display pipeline metrics:

Debezium Server

Each pipeline runs as a Debezium Server instance that exports metrics via OpenTelemetry (OTLP protocol).

OpenTelemetry Collector

Receives metrics from all Debezium Server instances and exports them in Prometheus format.

Prometheus

Scrapes metrics from the OpenTelemetry Collector and stores them as time-series data.

Conductor API

Queries Prometheus to retrieve metrics and serves them to the Stage UI through a REST API. The Conductor substitutes pipeline-specific identifiers into predefined queries and enforces access control.

Stage UI

Displays interactive monitoring dashboards with configurable time ranges and auto-refresh.

Monitoring dashboard

To access the monitoring dashboard for a pipeline, navigate to the pipeline detail page and click the Monitoring tab.

The monitoring dashboard provides the following controls:

Time range

Select the time window for the displayed metrics. Available presets include 5 minutes, 15 minutes, 30 minutes, 1 hour, 6 hours, 12 hours, and 24 hours. You can also specify a custom time range with specific start and end timestamps.

Refresh interval

Configure how often the dashboard automatically refreshes its data. Available options include off (manual refresh), 15 seconds, 30 seconds, 1 minute, and 5 minutes.

Built-in monitoring panels

The platform ships with a set of built-in monitoring panels organized into the following categories: streaming and snapshot.

Streaming panels

The streaming panels display metrics about the ongoing change data capture process.

Snapshot panels

The snapshot panels display metrics about the initial snapshot process when a pipeline first starts or when a snapshot is triggered.

The following image shows the snapshot panels while a snapshot is in progress:

The following image shows the snapshot panels after the snapshot has completed:

Prerequisites

Before you enable monitoring in the Debezium Platform, install the following operators in your Kubernetes cluster.

OpenTelemetry Operator (required)

The OpenTelemetry Operator manages the OpenTelemetryCollector custom resource that the platform Helm chart creates.

Install the operator by using Helm:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
  -n opentelemetry-operator-system --create-namespace \
  --set admissionWebhooks.certManager.enabled=false \
  --set admissionWebhooks.autoGenerateCert.enabled=true

The OpenTelemetry Collector must include the Prometheus exporter component. The base otelcol distribution includes this component, but when the OpenTelemetry Operator is installed by using Helm, it defaults to the otelcol-k8s distribution, which does not include the Prometheus exporter.

If your operator uses the k8s distribution, override the collector image by setting monitoring.otel.collector.image in your Helm values. For example:

• Base distribution: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:<version>
• Contrib distribution: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:<version>

See the opentelemetry-collector-releases repository for available distributions and their included components.

Prometheus Operator (optional)

The Prometheus Operator is required only if you want the Helm chart to create a ServiceMonitor for automatic Prometheus scraping. It is commonly installed through the kube-prometheus-stack chart:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

If you already have a Prometheus instance that does not use the Prometheus Operator, see Using an external Prometheus instance.

Enabling monitoring

To enable the monitoring infrastructure, set monitoring.otel.enabled to true and provide the Prometheus URL in your Helm values.

Procedure

1. Create a values file or add the following to your existing Helm values:

   monitoring:
     otel:
       enabled: true
       collector:
         # Override the collector image if needed (see Prerequisites)
         image: ""
     prometheus:
       url: "http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090"  (1)
       serviceMonitor:
         enabled: true
         labels:
           prometheus: kube-prometheus

   1	Required. Set the URL to match your Prometheus installation. The example assumes kube-prometheus-stack installed in the monitoring namespace with default settings.

2. Install or upgrade the platform with the monitoring values:

   helm upgrade --install debezium-platform debezium/debezium-platform \
     --set database.enabled=true \
     --set domain.name=platform.debezium.io \
     -f monitoring-values.yaml

When monitoring is enabled, the Helm chart automatically:

• Creates an OpenTelemetryCollector custom resource that is configured with OTLP receivers, a batch processor, and a Prometheus exporter.
• Creates a ServiceMonitor (if monitoring.prometheus.serviceMonitor.enabled is true) for automatic Prometheus scraping.
• Configures each pipeline’s Debezium Server to export metrics to the OpenTelemetry Collector.

Monitoring configuration reference

The following table lists the Helm chart properties for configuring monitoring.

Custom monitoring panels

You can extend the built-in monitoring dashboard by adding custom panels or overriding existing ones. Custom panels are defined in a YAML file and mounted into the Conductor container.

Panel definition format

Each panel is defined with the following fields:

panels:
  - id: my-custom-panel           (1)
    title: "Custom Metric"         (2)
    description: "Description"     (3)
    category: streaming            (4)
    query: 'rate(my_metric{service_name="{{pipeline_id}}"}[5m])'  (5)
    unit: ops/s                    (6)
    visualization:                 (7)
      type: line
      suggestedStep: 15s

Adding custom panels

Procedure

1. Create a YAML file that contains your panel definitions (for example, panels.yml).

2. Create a Kubernetes ConfigMap from the file:

   kubectl create configmap custom-panels \
     --from-file=panels.yml \
     -n debezium-platform

3. Configure the Helm chart to mount the ConfigMap and set the panels path.

   You can mount the ConfigMap in two ways:

   Option 1: Directory mount (recommended)

   Mount the ConfigMap as a directory. Kubernetes automatically propagates ConfigMap updates to the mounted volume, and the Conductor periodically reloads the panels file without requiring a pod restart.

   conductor:
     extraVolumes:
       - name: custom-panels
         configMap:
           name: custom-panels
     extraVolumeMounts:
       - name: custom-panels
         mountPath: /opt/config
         readOnly: true
   
   monitoring:
     panels:
       additionalPanelsPath: /opt/config/panels.yml

   Option 2: subPath mount

   Mount only the specific file by using subPath. This approach avoids creating extra files in the mount directory, but Kubernetes does not automatically update subPath mounts when the ConfigMap changes. You must restart the Conductor pod to pick up changes.

   conductor:
     extraVolumes:
       - name: custom-panels
         configMap:
           name: custom-panels
     extraVolumeMounts:
       - name: custom-panels
         mountPath: /opt/config/panels.yml
         subPath: panels.yml
         readOnly: true
   
   monitoring:
     panels:
       additionalPanelsPath: /opt/config/panels.yml

4. Upgrade the Helm release to apply the changes:

   helm upgrade debezium-platform debezium/debezium-platform \
     -n debezium-platform -f values.yaml

The Conductor automatically merges custom panels with the built-in defaults. Panels with matching id values override the corresponding built-in panels.

Using an external Prometheus instance

If you already have a Prometheus instance running in your cluster and you do not use the Prometheus Operator, you can point the platform to your existing Prometheus and disable the ServiceMonitor.

Procedure

1. Set monitoring.prometheus.url to the URL of your Prometheus instance and disable the ServiceMonitor:

   monitoring:
     otel:
       enabled: true
     prometheus:
       url: "http://prometheus.monitoring.svc.cluster.local:9090"
       serviceMonitor:
         enabled: false

2. Add a scrape configuration to your Prometheus configuration to scrape the OpenTelemetry Collector:

   scrape_configs:
     - job_name: 'otel-collector'
       scrape_interval: 15s
       static_configs:
         - targets: ['debezium-platform-otel-collector-collector.<namespace>.svc.cluster.local:8889']

   Replace <namespace> with the namespace where the Debezium Platform is deployed.