2. Monitoring Kyuubi - Server Metrics

Kyuubi has a configurable metrics system based on the Dropwizard Metrics Library. This allows users to report Kyuubi metrics to a variety of kyuubi.metrics.reporters. The metrics provide instrumentation for specific activities and Kyuubi server.

2.1. Configurations

The metrics system is configured via $KYUUBI_HOME/conf/kyuubi-defaults.conf.

Key Default Meaning Type Since

kyuubi.metrics.enabled

true

Set to true to enable kyuubi metrics system

boolean

1.2.0

kyuubi.metrics.reporters

JSON

A comma-separated list for all metrics reporters

• CONSOLE - ConsoleReporter which outputs measurements to CONSOLE periodically.
• JMX - JmxReporter which listens for new metrics and exposes them as MBeans.
• JSON - JsonReporter which outputs measurements to json file periodically.
• PROMETHEUS - PrometheusReporter which exposes metrics in Prometheus format.
• SLF4J - Slf4jReporter which outputs measurements to system log periodically.

seq

1.2.0

kyuubi.metrics.console.interval

PT5S

How often should report metrics to console

duration

1.2.0

kyuubi.metrics.json.interval

PT5S

How often should report metrics to JSON file

duration

1.2.0

kyuubi.metrics.json.location

metrics

Where the JSON metrics file located

string

1.2.0

kyuubi.metrics.prometheus.path

/metrics

URI context path of prometheus metrics HTTP server

string

1.2.0

kyuubi.metrics.prometheus.port

10019

Prometheus metrics HTTP server port

int

1.2.0

kyuubi.metrics.slf4j.interval

PT5S

How often should report metrics to SLF4J logger

duration

1.2.0

2.2. Metrics

These metrics include:

Metrics Prefix Metrics Suffix Type Since Description

kyuubi.exec.pool.threads.alive gauge 1.2.0

threads keepAlive in the backend executive thread pool

kyuubi.exec.pool.threads.active gauge 1.2.0

threads active in the backend executive thread pool

kyuubi.exec.pool.work_queue.size gauge 1.7.0

work queue size in the backend executive thread pool

kyuubi.connection.total counter 1.2.0

cumulative connection count

kyuubi.connection.total ${sessionType} counter 1.7.0

cumulative connection count with session type ${sessionType}

kyuubi.connection.opened gauge 1.2.0

current active connection count

kyuubi.connection.opened ${user} counter 1.2.0

current active connections count requested by a ${user}

kyuubi.connection.opened ${user}
${sessionType} counter 1.7.0

current active connections count requested by a ${user} with session type ${sessionType}

kyuubi.connection.opened ${sessionType} counter 1.7.0

current active connections count with session type ${sessionType}

kyuubi.connection.failed counter 1.2.0

cumulative failed connection count

kyuubi.connection.failed ${user} counter 1.2.0

cumulative failed connections for a ${user}

kyuubi.connection.failed ${sessionType} counter 1.7.0

cumulative failed connection count with session type ${sessionType}

kyuubi.operation.total counter 1.5.0

cumulative opened operation count

kyuubi.operation.total ${operationType} counter 1.5.0

cumulative opened count for the operation ${operationType}

kyuubi.operation.opened gauge 1.5.0

current opened operation count

kyuubi.operation.opened ${operationType} counter 1.5.0

current opened count for the operation ${operationType}

kyuubi.operation.failed ${operationType}
.${errorType} counter 1.5.0

cumulative failed count for the operation ${operationType} with a particular ${errorType}, e.g. execute_statement.AnalysisException

kyuubi.operation.state ${operationState} meter 1.5.0

kyuubi operation state rate

kyuubi.operation.exec_time ${operationType} histogram 1.7.0

execution time histogram for the operation ${operationType}, now only ExecuteStatement is enabled.

kyuubi.engine.total counter 1.2.0

cumulative created engines

kyuubi.engine.timeout counter 1.2.0

cumulative timeout engines

kyuubi.engine.failed ${user} counter 1.2.0

cumulative explicitly failed engine count for a ${user}

kyuubi.engine.failed ${errorType} counter 1.2.0

cumulative explicitly failed engine count for a particular ${errorType}, e.g. ClassNotFoundException

kyuubi.backend_service.open_session timer 1.5.0

kyuubi backend service openSession method execution time and rate

kyuubi.backend_service.close_session timer 1.5.0

kyuubi backend service closeSession method execution time and rate

kyuubi.backend_service.get_info timer 1.5.0

kyuubi backend service getInfo method execution time and rate

kyuubi.backend_service.execute_statement timer 1.5.0

kyuubi backend service executeStatement method execution time and rate

kyuubi.backend_service.get_type_info timer 1.5.0

kyuubi backend service getTypeInfo method execution time and rate

kyuubi.backend_service.get_catalogs timer 1.5.0

kyuubi backend service getCatalogs method execution time and rate

kyuubi.backend_service.get_schemas timer 1.5.0

kyuubi backend service getSchemas method execution time and rate

kyuubi.backend_service.get_tables timer 1.5.0

kyuubi backend service getTables method execution time and rate

kyuubi.backend_service.get_table_types timer 1.5.0

kyuubi backend service getTableTypes method execution time and rate

kyuubi.backend_service.get_columns timer 1.5.0

kyuubi backend service getColumns method execution time and rate

kyuubi.backend_service.get_functions timer 1.5.0

kyuubi backend service getFunctions method execution time and rate

kyuubi.backend_service.get_operation_status timer 1.5.0

kyuubi backend service getOperationStatus method execution time and rate

kyuubi.backend_service.cancel_operation timer 1.5.0

kyuubi backend service cancelOperation method execution time and rate

kyuubi.backend_service.close_operation timer 1.5.0

kyuubi backend service closeOperation method execution time and rate

kyuubi.backend_service.get_result_set_metadata timer 1.5.0

kyuubi backend service getResultSetMetadata method execution time and rate

kyuubi.backend_service.fetch_results timer 1.5.0

kyuubi backend service fetchResults method execution time and rate

kyuubi.backend_service.fetch_log_rows_rate meter 1.5.0

kyuubi backend service fetchResults method that fetch log rows rate

kyuubi.backend_service.fetch_result_rows_rate meter 1.5.0

kyuubi backend service fetchResults method that fetch result rows rate

kyuubi.backend_service.get_primary_keys meter 1.6.0

kyuubi backend service get_primary_keys method execution time and rate

kyuubi.backend_service.get_cross_reference meter 1.6.0

kyuubi backend service get_cross_reference method execution time and rate

kyuubi.operation.state ${operationType}
.${state} meter 1.6.0

The ${operationType} with a particular ${state} rate, e.g. BatchJobSubmission.pending, BatchJobSubmission.finished. Note that, the terminal states are cumulative, but the intermediate ones are not.

kyuubi.metadata.request.opened counter 1.6.1

current opened count for the metadata requests

kyuubi.metadata.request.total meter 1.6.0

metadata requests time and rate

kyuubi.metadata.request.failed meter 1.6.0

metadata requests failure time and rate

kyuubi.metadata.request.retrying meter 1.6.0

retrying metadata requests time and rate, it is not cumulative

Before v1.5.0, if you use these metrics:

• kyuubi.statement.total
• kyuubi.statement.opened
• kyuubi.statement.failed.${errorType}

Since v1.5.0, you can use the following metrics to replace:

• kyuubi.operation.total.ExecuteStatement
• kyuubi.operation.opened.ExecuteStatement
• kyuubi.operation.failed.ExecuteStatement.${errorType}