Debezium logging

Debezium has extensive logging built into its connectors, and you can change the logging configuration to control which of these log statements appear in the logs and where those logs are sent. Debezium (as well as Kafka and Kafka Connect) use the Log4j logging framework for Java.

By default, the connectors produce a fair amount of useful information when they start up, but then produce very few logs when the connector is keeping up with the source databases. This is often sufficient when the connector is operating normally, but may not be enough when the connector is behaving unexpectedly. In such cases, you can change the logging level so that the connector generates much more verbose log messages describing what the connector is doing and what it is not doing.

Logging concepts

The Log4j logging configuration determines which Debezium messages you see and where they appear, so that you can monitor connector health, troubleshoot problems, and understand connector behavior. Understanding how loggers, log levels, and appenders work together helps you to set a logging configuration that filters log output in ways that enhance visibility of relevant information.

Loggers

Debezium sends each log message that it produces to a specific logger, for example, io.debezium.connector.mysql. Loggers are arranged in hierarchies. For example, the io.debezium.connector.mysql logger is the child of the io.debezium.connector logger, which is the child of the io.debezium logger. The root logger is the parent of all other loggers in the hierarchy and its configuration serves as the default configuration for all of its descendants. You can override the default configuration of descendant loggers to specify unique log levels and appenders.

Log levels

The application assigns one of the following log levels to each log message:

Appenders

An appender specifies the destination where Debezium writes log messages. Because each appender delegates message formatting to an associated Layout, you can control the message format by sending messages to a specific appender.

Understanding the default logging configuration

If you are running Debezium connectors in a Kafka Connect process, then Kafka Connect uses the Log4j configuration file (for example, /opt/kafka/config/connect-log4j.properties) in the Kafka installation. The following block shows an example of a default configuration in connect-log4j.properties:

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n
...

The following list describes select fields in the preceding connect-log4j.properties configuration:

log4j.rootLogger

The root logger, which defines the default logger configuration. By default, loggers include INFO, WARN, and ERROR messages. These log messages are written to the stdout appender.

log4j.appender.stdout

Directs the stdout appender to write log messages to the console, as opposed to a file.

log4j.appender.stdout.layout

Specifies that the stdout appender uses a pattern matching algorithm to format log messages.

log4j.appender.stdout.layout.ConversionPattern

The pattern that the stdout appender uses (see the Log4j documentation for details).

Unless you configure other loggers, all of the loggers that Debezium uses inherit the rootLogger configuration.

Configuring logging

By default, Debezium connectors write all INFO, WARN, and ERROR messages to the console. You can change the default logging configuration by using one of the following methods:

• Setting the logging level by configuring loggers
• Dynamically setting the logging level with the Kafka Connect REST API
• Setting the logging level by adding mapped diagnostic contexts

Setting the logging level by configuring loggers

The default Debezium logging level provides sufficient information to show whether a connector is healthy or not. However, if a connector is not healthy, you can change its logging level to troubleshoot the issue.

In general, Debezium connectors send their log messages to loggers with names that match the fully-qualified name of the Java class that is generating the log message. Debezium uses packages to organize code with similar or related functions. This means that you can control all of the log messages for a specific class or for all of the classes within or under a specific package.

Procedure

1. Open the connect-log4j.properties file.

2. Configure a logger for the connector.

   The following example shows a possible connect-log4j.properties configuration to enable loggers for a MySQL connector to monitor the connector’s database schema history implementation at a DEBUG log level:

   ...
   log4j.logger.io.debezium.connector.mysql=DEBUG, stdout
   log4j.logger.io.debezium.relational.history=DEBUG, stdout
   
   log4j.additivity.io.debezium.connector.mysql=false
   log4j.additivity.io.debezium.storage.kafka.history=false
   ...

   The following list describes select fields in the preceding connect-log4j.properties configuration:

   log4j.logger.io.debezium.connector.mysql

   Configures the logger named io.debezium.connector.mysql to send DEBUG, INFO, WARN, and ERROR messages to the stdout appender.

   log4j.logger.io.debezium.relational.history

   Configures the logger named io.debezium.relational.history to send DEBUG, INFO, WARN, and ERROR messages to the stdout appender.

   log4j.additivity.io.debezium.connector.mysql

   This pair of log4j.additivity.io entries disable additivity. If you use multiple appenders, set additivity values to false to prevent duplicate log messages from being sent to the appenders of the parent loggers.

3. If necessary, change the logging level for a specific subset of the classes within the connector.

   Increasing the logging level for the entire connector increases the log verbosity, which can make it difficult to understand what is happening. In these cases, you can change the logging level just for the subset of classes that are related to the issue that you are troubleshooting.

   1. Set the connector’s logging level to either DEBUG or TRACE.

   2. Review the connector’s log messages.

      Find the log messages that are related to the issue that you are troubleshooting. The end of each log message shows the name of the Java class that produced the message.

   3. Set the connector’s logging level back to INFO.

   4. Configure a logger for each Java class that you identified.

      For example, consider a scenario in which you are unsure why the MySQL connector is skipping some events when it is processing the binlog. Rather than turn on DEBUG or TRACE logging for the entire connector, you can keep the connector’s logging level at INFO and then configure DEBUG or TRACE on just the class that is reading the binlog:

      Example 1. connect-log4j.properties configuration that enables DEBUG logging for the BinlogReader class

      ...
      log4j.logger.io.debezium.connector.mysql=INFO, stdout
      log4j.logger.io.debezium.connector.mysql.BinlogReader=DEBUG, stdout
      log4j.logger.io.debezium.relational.history=INFO, stdout
      
      log4j.additivity.io.debezium.connector.mysql=false
      log4j.additivity.io.debezium.storage.kafka.history=false
      log4j.additivity.io.debezium.connector.mysql.BinlogReader=false
      ...

Dynamically setting the logging level with the Kafka Connect REST API

You can use the Kafka Connect REST API to set logging levels for a connector dynamically at runtime. Unlike log level changes that you set in connect-log4j.properties, changes that you make via the API take effect immediately, and do not require you to restart the worker.

The log level setting that you specify in the API applies only to the worker at the endpoint that receives the request. The log levels of other workers in the cluster remain unchanged.

The specified level is not persisted after the worker restarts. To make persistent changes to the logging level, set the log level in connect-log4j.properties by configuring loggers or adding mapped diagnostic contexts.

Procedure

• Set the log level by sending a PUT request to the admin/loggers endpoint that specifies the following information:

  • The package for which you want to change the log level.

  • The log level that you want to set.

    curl -s -X PUT -H "Content-Type:application/json" http://localhost:8083/admin/loggers/io.debezium.connector.<connector_package> -d '{"level": "<log_level>"}'

    For example, to log debug information for a Debezium MySQL connector, send the following request to Kafka Connect:

    curl -s -X PUT -H "Content-Type:application/json" http://localhost:8083/admin/loggers/io.debezium.connector.mysql -d '{"level": "DEBUG"}'

Setting the logging level with mapped diagnostic contexts

Most Debezium connectors (and the Kafka Connect workers) use multiple threads to perform different activities. This can make it difficult to look at a log file and find only those log messages for a particular logical activity. To make the log messages easier to find, Debezium provides several mapped diagnostic contexts (MDC) that provide additional information for each thread.

Debezium provides the following MDC properties:

dbz.connectorType

A short alias for the type of connector. For example, MySql, Mongo, Postgres, and so on. All threads associated with the same type of connector use the same value, so you can use this to find all log messages produced by a given type of connector.

dbz.connectorLogicalName

The name of the connector or database server as defined in the connector’s configuration. For example products, serverA, and so on. All threads associated with a specific connector instance use the same value, so you can find all of the log messages produced by a specific connector instance.

dbz.connectorContext

A short name for an activity running as a separate thread running within the connector’s task. For example, main, binlog, snapshot, and so on. In some cases, when a connector assigns threads to specific resources (such as a table or collection), the name of that resource could be used instead. Each thread associated with a connector would use a distinct value, so you can find all of the log messages associated with this particular activity.

To enable MDC for a connector, you configure an appender in the connect-log4j.properties file.

Procedure

1. Open the connect-log4j.properties file.

2. Configure an appender to use any of the supported Debezium MDC properties. In the following example, the stdout appender is configured to use these MDC properties.

   Example 2. connect-log4j.properties configuration that sets the stdout appender to use MDC properties

   ...
   log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p  %X{dbz.connectorType}|%X{dbz.connectorLogicalName}|%X{dbz.connectorContext}  %m   [%c]%n
   ...

   The configuration in the preceding example produces log messages similar to the ones in the following output:

   ...
   2017-02-07 20:49:37,692 INFO   MySQL|dbserver1|snapshot  Starting snapshot for jdbc:mysql://mysql:3306/?useInformationSchema=true&nullCatalogMeansCurrent=false&useSSL=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=convertToNull with user 'debezium'   [io.debezium.connector.mysql.SnapshotReader]
   2017-02-07 20:49:37,696 INFO   MySQL|dbserver1|snapshot  Snapshot is using user 'debezium' with these MySQL grants:   [io.debezium.connector.mysql.SnapshotReader]
   2017-02-07 20:49:37,697 INFO   MySQL|dbserver1|snapshot  GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'debezium'@'%'   [io.debezium.connector.mysql.SnapshotReader]
   ...

Each line in the log includes the connector type (for example, MySQL), the name of the connector (for example, dbserver1), and the activity of the thread (for example, snapshot).

Configuring the log level in the Debezium container images

The Debezium container images for Kafka and Kafka Connect all set up their log4j.properties file to configure the Debezium-related loggers. All log messages are sent to the Docker container’s console (and thus the Docker logs). The log messages are also written to files under the /kafka/logs directory.

The containers use a LOG_LEVEL environment variable to set the log level for the root logger. You can use this environment variable to set the log level for the service running in the container. When you start the container and set the value of this environment variable to a log level (for example, -e LOG_LEVEL=DEBUG), all of the code within the container then uses that log level.

There is also an option to override other log4j properties. If you want to configure log4j.rootLogger differently, then use the environment variable CONNECT_LOG4J_LOGGERS. For example to log only to stdout (without appender), you can use CONNECT_LOG4J_LOGGERS=INFO, stdout. You can also set other supported log4j environment variables with the CONNECT_LOG4J prefix, which will be mapped to properties in the log4j.properties file by removing the CONNECT_ prefix, lowercasing all characters, and converting all '_' characters to '.'.

If you need more control over the logging configuration, create a new container image that is based on ours, except that in your Dockerfile, copy your own log4j.properties file into the image. For example:

Dockerfile

...
COPY log4j.properties $KAFKA_HOME/config/log4j.properties
...