Kafka Metrics¶

Introduction¶

This section describes metrics reported by DMS to Cloud Eye as well as their namespaces and dimensions. You can use the Cloud Eye console or APIs to query the Kafka metrics and alarms, or view Kafka instance metrics on the Monitoring page of the DMS console.

For example, you can call the API to query the monitoring data of the Disk Capacity Usage metric.

Namespace¶

SYS.DMS

Instance Metrics¶

**Table 1** Instance metrics¶
Metric ID	Metric Name	Description	Value Range	Monitored Object	Monitoring Period (Raw Data)
current_partitions	Partitions	Number of used partitions in the instance Unit: count	0-100,000	Kafka instance	1 minute
current_topics	Topics	Number of created topics in the instance Unit: count	0-100,000	Kafka instance	1 minute
group_msgs	Accumulated Messages	Total number of accumulated messages in all consumer groups of the instance Unit: count	0-1,000,000,000	Kafka instance	1 minute

Broker Metrics¶

Enabling Smart Connect for a Kafka instance creates two or more brokers. On the By Broker tab page, select "connector" for Node Type for Smart Connect broker metrics, or select "broker" for Node Type for Kafka instance broker metrics.

Metrics of Smart Connect brokers: disk capacity usage, memory usage, JVM heap memory usage, node alive status, and connections.

**Table 2** Broker metrics¶
Metric ID	Metric Name	Description	Value Range	Monitored Object	Monitoring Period (Raw Data)
broker_data_size	Message Size	Total size of messages in the broker Unit: byte, KB, MB, GB, TB or PB	0-5,000,000,000,000	Kafka instance broker	1 minute
broker_messages_in_rate	Message Creation Rate	Number of messages created per second Unit: count/s	0-500,000	Kafka instance broker	1 minute
broker_bytes_out_rate	Message Retrieval	Number of bytes retrieved per second Unit: byte/s, KB/s, MB/s, or GB/s	0-500,000,000	Kafka instance broker	1 minute
broker_bytes_in_rate	Message Creation	Number of bytes created per second Unit: byte/s, KB/s, MB/s, or GB/s	0-500,000,000	Kafka instance broker	1 minute
broker_public_bytes_in_rate	Public Inbound Traffic	Inbound traffic over public networks per second Unit: byte/s, KB/s, MB/s, or GB/s Note You can view this metric on the EIP console if public access has been enabled and EIPs have been assigned to the instance.	0-500,000,000	Kafka instance broker	1 minute
broker_public_bytes_out_rate	Public Outbound Traffic	Outbound traffic over public networks per second Unit: byte/s, KB/s, MB/s, or GB/s Note You can view this metric on the EIP console if public access has been enabled and EIPs have been assigned to the instance.	0-500,000,000	Kafka instance broker	1 minute
broker_fetch_mean	Average Message Retrieval Processing Duration	Average time that the broker spends processing message retrieval requests Unit: ms	0-10,000	Kafka instance broker	1 minute
broker_produce_mean	Average Message Creation Processing Duration	Average time that the broker spends processing message creation requests Unit: ms	0-10,000	Kafka instance broker	1 minute
broker_cpu_core_load	Average Load per CPU Core	Average load of each CPU core of the Kafka VM Unit: %	0-20	Kafka instance broker	1 minute
broker_disk_usage	Disk Capacity Usage	Disk usage of the Kafka VM Unit: %	0-100	Kafka instance broker	1 minute
broker_memory_usage	Memory Usage	Memory usage of the Kafka VM Unit: %	0-100	Kafka instance broker	1 minute
broker_heap_usage	JVM Heap Memory Usage of Kafka	Heap memory usage of the Kafka JVM Unit: %	0-100	Kafka instance broker	1 minute
broker_alive	Broker Alive	Whether the Kafka broker is alive	1: alive 0: not alive	Kafka instance broker	1 minute
broker_connections	Connections	Total number of TCP connections on the Kafka broker Unit: count	0-65,535	Kafka instance broker	1 minute
broker_cpu_usage	CPU Usage	CPU usage of the Kafka VM Unit: %	0-100	Kafka instance broker	1 minute
broker_disk_read_await	Average Disk Read Time	Average time for each disk I/O read in the monitoring period Unit: ms	> 0	Kafka instance broker	1 minute
broker_disk_write_await	Average Disk Write Time	Average time for each disk I/O write in the monitoring period Unit: ms	> 0	Kafka instance broker	1 minute
broker_total_bytes_in_rate	Inbound Traffic	Inbound traffic per second Unit: byte/s	0-1,000,000,000	Kafka instance broker	1 minute
broker_total_bytes_out_rate	Outbound Traffic	Outbound traffic per second Unit: byte/s	0-1,000,000,000	Kafka instance broker	1 minute
broker_disk_read_rate	Disk Read Speed	Read traffic on the disk Unit: byte/s, KB/s, MB/s, or GB/s	>= 0	Kafka instance broker	1 minute
broker_disk_write_rate	Disk Write Speed	Write traffic on the disk Unit: byte/s, KB/s, MB/s, or GB/s	>= 0	Kafka instance broker	1 minute

Topic Metrics¶

**Table 3** Topic metrics¶
Metric ID	Metric Name	Description	Value Range	Monitored Object	Monitoring Period (Raw Data)
topic_bytes_in_rate	Message Creation	Number of bytes created per second Unit: byte/s, KB/s, MB/s, or GB/s Note On the Cloud Eye console, this metric is available only when Scope is set to Basic monitoring on the Queues tab page. On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.	0-500,000,000	Topic in a Kafka instance	1 minute
topic_bytes_out_rate	Message Retrieval	Number of bytes retrieved per second Unit: byte/s, KB/s, MB/s, or GB/s Note On the Cloud Eye console, this metric is available only when Scope is set to Basic monitoring on the Queues tab page. On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.	0-500,000,000	Topic in a Kafka instance	1 minute
topic_data_size	Message Size	Total size of messages in the queue Unit: byte, KB, MB, GB, TB or PB Note On the Cloud Eye console, this metric is available only when Scope is set to Basic monitoring on the Queues tab page. On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.	0-5,000,000,000,000	Topic in a Kafka instance	1 minute
topic_messages	Total Messages	Total number of messages in the queue Unit: count Note On the Cloud Eye console, this metric is available only when Scope is set to Basic monitoring on the Queues tab page. On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.	>= 0	Topic in a Kafka instance	1 minute
topic_messages_in_rate	Message Creation Rate	Number of messages created per second Unit: count/s Note On the Cloud Eye console, this metric is available only when Scope is set to Basic monitoring on the Queues tab page. On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Basic monitoring on the By Topic tab page.	0-500,000	Topic in a Kafka instance	1 minute
partition_messages	Partition Messages	Total number of messages in the partition Unit: count Note On the Cloud Eye console, this metric is available only when Scope is set to Partition monitoring on the Queues tab page. On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page.	>= 0	Topic in a Kafka instance	1 minute
produced_messages	Created Messages	Number of messages that have been created Unit: count Note On the Cloud Eye console, this metric is available only when Scope is set to Partition monitoring on the Queues tab page. On the Monitoring page of the DMS console, this metric is available only when Monitoring Type is set to Partition monitoring on the By Topic tab page.	>= 0	Topic in a Kafka instance	1 minute

Consumer Group Metrics¶

**Table 4** Consumer group metrics¶
Metric ID	Metric Name	Description	Value Range	Monitored Object	Monitoring Period (Raw Data)
messages_consumed	Retrieved Messages	Number of messages that have been retrieved in the consumer group Unit: count Note On the Cloud Eye console, this metric is available only when Queue is set to a specific topic name and Scope is set to Partition monitoring on the Consumer Groups tab page. On the Monitoring page of the DMS console, this metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.	>= 0	Consumer group of a Kafka instance	1 minute
messages_remained	Available Messages	Number of messages that can be retrieved in the consumer group Unit: count Note On the Cloud Eye console, this metric is available only when Queue is set to a specific topic name and Scope is set to Partition monitoring on the Consumer Groups tab page. On the Monitoring page of the DMS console, this metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Partition monitoring on the By Consumer Group tab page.	>= 0	Consumer group of a Kafka instance	1 minute
topic_messages_remained	Topic Available Messages	Number of remaining messages that can be retrieved from the specified topic in the consumer group Unit: Count Note On the Cloud Eye console, this metric is available only when Queue is set to a specific topic name and Scope is set to Basic monitoring on the Consumer Groups tab page. On the Monitoring page of the DMS console, this metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.	0 to 2⁶³-1	Consumer group of a Kafka instance	1 minute
topic_messages_consumed	Topic Retrieved Messages	Number of messages that have been retrieved from the specified topic in the consumer group Unit: Count Note On the Cloud Eye console, this metric is available only when Queue is set to a specific topic name and Scope is set to Basic monitoring on the Consumer Groups tab page. On the Monitoring page of the DMS console, this metric is available only when Topic is set to a specific topic name and Monitoring Type is set to Basic monitoring on the By Consumer Group tab page.	0 to 2⁶³-1	Consumer group of a Kafka instance	1 minute
consumer_messages_remained	Consumer Available Messages	Number of remaining messages that can be retrieved in the consumer group Unit: Count Note On the Cloud Eye console, this metric is available only when Queues is set to All queues on the Consumer Groups tab page. On the Monitoring page of the DMS console, this metric is available only when Topic is set to All topics on the By Consumer Group tab page.	0 to 2⁶³-1	Consumer group of a Kafka instance	1 minute
consumer_messages_consumed	Consumer Retrieved Messages	Number of messages that have been retrieved in the consumer group Unit: Count Note On the Cloud Eye console, this metric is available only when Queues is set to All queues on the Consumer Groups tab page. On the Monitoring page of the DMS console, this metric is available only when Topic is set to All topics on the By Consumer Group tab page.	0 to 2⁶³-1	Consumer group of a Kafka instance	1 minute

Smart Connect Metrics¶

**Table 5** Smart Connect metrics¶
Metric ID	Metric Name	Description	Value Range	Monitored Object	Monitoring Period (Raw Data)
kafka_wait_synchronize_data	Kafka Data to Sync	Data to synchronize in the Kafka migration task Unit: count	>= 0	Smart Connect task of a Kafka instance	1 minute
kafka_synchronize_rate	Kafka Data Synced per Minute	Data synchronized per minute in the Kafka migration task Unit: count	>= 0	Smart Connect task of a Kafka instance	1 minute
task_status	Task Status	Status of the current task	0: abnormal 1: normal	Smart Connect task of a Kafka instance	1 minute
message_delay	Message Delay	Time elapsed between when a message is sent from the source and received by the target Unit: ms	>= 0	Smart Connect task of a Kafka instance	1 minute

Note

A Smart Connect task that bidirectionally copies Kafka data is split into two tasks for monitoring: Smart Connect task name_source_0 and Smart Connect task name_source_1.
If all messages in a topic have aged before the next synchronization, there is no Kafka data to be synchronized. However, since the Kafka data synchronization metric uses the offset value that contains aged data, Kafka Data Synced per Minute will display the number of aged messages.

Dimension¶

Key	Value
kafka_instance_id	Kafka instance
kafka_broker	Kafka instance broker
kafka_topics	Kafka instance topic
kafka_partitions	Partition in a Kafka instance
kafka_groups-partitions	Partition consumer group in a Kafka instance
kafka_groups_topics	Topic consumer group in a Kafka instance
kafka_groups	Consumer group of a Kafka instance

last updated: 2024-11-13 14:47 UTC - commit: bab3492e4e325ec745b7b7fa1576930332ed8243