Kafka Metrics

Introduction

This section describes metrics reported by DMS to Cloud Eye as well as their namespaces and dimensions. You can use the Cloud Eye console or APIs to query the Kafka metrics and alarms, or view Kafka instance metrics on the Monitoring page of the DMS console.

For example, you can call the API to query the monitoring data of the Disk Capacity Usage metric.

Namespace

SYS.DMS

Instance Metrics

Table 1 Instance metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

current_partitions

Partitions

Number of used partitions in the instance

Unit: count

0-100,000

Kafka instance

1 minute

current_topics

Topics

Number of created topics in the instance

Unit: count

0-100,000

Kafka instance

1 minute

group_msgs

Accumulated Messages

Total number of accumulated messages in all consumer groups of the instance

Unit: count

0-1,000,000,000

Kafka instance

1 minute

Broker Metrics

Enabling Smart Connect for a Kafka instance creates two or more brokers. On the By Broker tab page, select "connector" for Node Type for Smart Connect broker metrics, or select "broker" for Node Type for Kafka instance broker metrics.

Metrics of Smart Connect brokers: disk capacity usage, memory usage, JVM heap memory usage, node alive status, and connections.

Table 2 Broker metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

broker_data_size

Message Size

Total size of messages in the broker

Unit: byte, KB, MB, GB, TB or PB

0-5,000,000,000,000

Kafka instance broker

1 minute

broker_messages_in_rate

Message Creation Rate

Number of messages created per second

Unit: count/s

0-500,000

Kafka instance broker

1 minute

broker_bytes_out_rate

Message Retrieval

Number of bytes retrieved per second

Unit: byte/s, KB/s, MB/s, or GB/s

0-500,000,000

Kafka instance broker

1 minute

broker_bytes_in_rate

Message Creation

Number of bytes created per second

Unit: byte/s, KB/s, MB/s, or GB/s

0-500,000,000

Kafka instance broker

1 minute

broker_public_bytes_in_rate

Public Inbound Traffic

Inbound traffic over public networks per second

Unit: byte/s, KB/s, MB/s, or GB/s

Note

You can view this metric on the EIP console if public access has been enabled and EIPs have been assigned to the instance.

0-500,000,000

Kafka instance broker

1 minute

broker_public_bytes_out_rate

Public Outbound Traffic

Outbound traffic over public networks per second

Unit: byte/s, KB/s, MB/s, or GB/s

Note

You can view this metric on the EIP console if public access has been enabled and EIPs have been assigned to the instance.

0-500,000,000

Kafka instance broker

1 minute

broker_fetch_mean

Average Message Retrieval Processing Duration

Average time that the broker spends processing message retrieval requests

Unit: ms

0-10,000

Kafka instance broker

1 minute

broker_produce_mean

Average Message Creation Processing Duration

Average time that the broker spends processing message creation requests

Unit: ms

0-10,000

Kafka instance broker

1 minute

broker_cpu_core_load

Average Load per CPU Core

Average load of each CPU core of the Kafka VM

Unit: %

0-20

Kafka instance broker

1 minute

broker_disk_usage

Disk Capacity Usage

Disk usage of the Kafka VM

Unit: %

0-100

Kafka instance broker

1 minute

broker_memory_usage

Memory Usage

Memory usage of the Kafka VM

Unit: %

0-100

Kafka instance broker

1 minute

broker_heap_usage

JVM Heap Memory Usage of Kafka

Heap memory usage of the Kafka JVM

Unit: %

0-100

Kafka instance broker

1 minute

broker_alive

Broker Alive

Whether the Kafka broker is alive

  • 1: alive

  • 0: not alive

Kafka instance broker

1 minute

broker_connections

Connections

Total number of TCP connections on the Kafka broker

Unit: count

0-65,535

Kafka instance broker

1 minute

broker_cpu_usage

CPU Usage

CPU usage of the Kafka VM

Unit: %

0-100

Kafka instance broker

1 minute

broker_disk_read_await

Average Disk Read Time

Average time for each disk I/O read in the monitoring period

Unit: ms

> 0

Kafka instance broker

1 minute

broker_disk_write_await

Average Disk Write Time

Average time for each disk I/O write in the monitoring period

Unit: ms

> 0

Kafka instance broker

1 minute

broker_total_bytes_in_rate

Inbound Traffic

Inbound traffic per second

Unit: byte/s

0-1,000,000,000

Kafka instance broker

1 minute

broker_total_bytes_out_rate

Outbound Traffic

Outbound traffic per second

Unit: byte/s

0-1,000,000,000

Kafka instance broker

1 minute

broker_disk_read_rate

Disk Read Speed

Read traffic on the disk

Unit: byte/s, KB/s, MB/s, or GB/s

>= 0

Kafka instance broker

1 minute

broker_disk_write_rate

Disk Write Speed

Write traffic on the disk

Unit: byte/s, KB/s, MB/s, or GB/s

>= 0

Kafka instance broker

1 minute

Topic Metrics

Table 3 Topic metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

topic_bytes_in_rate

Message Creation

Number of bytes created per second

Unit: byte/s, KB/s, MB/s, or GB/s

Note

  • On the Cloud Eye console, this metric is available only when Scope is set to Basic monitoring on the Queues tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.

0-500,000,000

Topic in a Kafka instance

1 minute

topic_bytes_out_rate

Message Retrieval

Number of bytes retrieved per second

Unit: byte/s, KB/s, MB/s, or GB/s

Note

  • On the Cloud Eye console, this metric is available only when Scope is set to Basic monitoring on the Queues tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.

0-500,000,000

Topic in a Kafka instance

1 minute

topic_data_size

Message Size

Total size of messages in the queue

Unit: byte, KB, MB, GB, TB or PB

Note

  • On the Cloud Eye console, this metric is available only when Scope is set to Basic monitoring on the Queues tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.

0-5,000,000,000,000

Topic in a Kafka instance

1 minute

topic_messages

Total Messages

Total number of messages in the queue

Unit: count

Note

  • On the Cloud Eye console, this metric is available only when Scope is set to Basic monitoring on the Queues tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.

>= 0

Topic in a Kafka instance

1 minute

topic_messages_in_rate

Message Creation Rate

Number of messages created per second

Unit: count/s

Note

  • On the Cloud Eye console, this metric is available only when Scope is set to Basic monitoring on the Queues tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.

0-500,000

Topic in a Kafka instance

1 minute

partition_messages

Partition Messages

Total number of messages in the partition

Unit: count

Note

  • On the Cloud Eye console, this metric is available only when Scope is set to Partition monitoring on the Queues tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page.

>= 0

Topic in a Kafka instance

1 minute

produced_messages

Created Messages

Number of messages that have been created

Unit: count

Note

  • On the Cloud Eye console, this metric is available only when Scope is set to Partition monitoring on the Queues tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page.

>= 0

Topic in a Kafka instance

1 minute

Consumer Group Metrics

Table 4 Consumer group metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

messages_consumed

Retrieved Messages

Number of messages that have been retrieved in the consumer group

Unit: count

Note

  • On the Cloud Eye console, this metric is available only when Queue is set to a specific topic name and Scope is set to Partition monitoring on the Consumer Groups tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.

>= 0

Consumer group of a Kafka instance

1 minute

messages_remained

Available Messages

Number of messages that can be retrieved in the consumer group

Unit: count

Note

  • On the Cloud Eye console, this metric is available only when Queue is set to a specific topic name and Scope is set to Partition monitoring on the Consumer Groups tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.

>= 0

Consumer group of a Kafka instance

1 minute

topic_messages_remained

Topic Available Messages

Number of remaining messages that can be retrieved from the specified topic in the consumer group

Unit: Count

Note

  • On the Cloud Eye console, this metric is available only when Queue is set to a specific topic name and Scope is set to Basic monitoring on the Consumer Groups tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.

0 to 263-1

Consumer group of a Kafka instance

1 minute

topic_messages_consumed

Topic Retrieved Messages

Number of messages that have been retrieved from the specified topic in the consumer group

Unit: Count

Note

  • On the Cloud Eye console, this metric is available only when Queue is set to a specific topic name and Scope is set to Basic monitoring on the Consumer Groups tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.

0 to 263-1

Consumer group of a Kafka instance

1 minute

consumer_messages_remained

Consumer Available Messages

Number of remaining messages that can be retrieved in the consumer group

Unit: Count

Note

  • On the Cloud Eye console, this metric is available only when Queues is set to All queues on the Consumer Groups tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Topic is set to All topics on the By Consumer Group tab page.

0 to 263-1

Consumer group of a Kafka instance

1 minute

consumer_messages_consumed

Consumer Retrieved Messages

Number of messages that have been retrieved in the consumer group

Unit: Count

Note

  • On the Cloud Eye console, this metric is available only when Queues is set to All queues on the Consumer Groups tab page.

  • On the Monitoring page of the DMS console, this metric is available only when Topic is set to All topics on the By Consumer Group tab page.

0 to 263-1

Consumer group of a Kafka instance

1 minute

Smart Connect Metrics

Table 5 Smart Connect metrics

Metric ID

Metric Name

Description

Value Range

Monitored Object

Monitoring Period (Raw Data)

kafka_wait_synchronize_data

Kafka Data to Sync

Data to synchronize in the Kafka migration task

Unit: count

>= 0

Smart Connect task of a Kafka instance

1 minute

kafka_synchronize_rate

Kafka Data Synced per Minute

Data synchronized per minute in the Kafka migration task

Unit: count

>= 0

Smart Connect task of a Kafka instance

1 minute

task_status

Task Status

Status of the current task

  • 0: abnormal

  • 1: normal

Smart Connect task of a Kafka instance

1 minute

message_delay

Message Delay

Time elapsed between when a message is sent from the source and received by the target

Unit: ms

>= 0

Smart Connect task of a Kafka instance

1 minute

Note

  • A Smart Connect task that bidirectionally copies Kafka data is split into two tasks for monitoring: Smart Connect task name_source_0 and Smart Connect task name_source_1.

  • If all messages in a topic have aged before the next synchronization, there is no Kafka data to be synchronized. However, since the Kafka data synchronization metric uses the offset value that contains aged data, Kafka Data Synced per Minute will display the number of aged messages.

Dimension

Key

Value

kafka_instance_id

Kafka instance

kafka_broker

Kafka instance broker

kafka_topics

Kafka instance topic

kafka_partitions

Partition in a Kafka instance

kafka_groups-partitions

Partition consumer group in a Kafka instance

kafka_groups_topics

Topic consumer group in a Kafka instance

kafka_groups

Consumer group of a Kafka instance