• MapReduce Service

mrs
  1. Help Center
  2. MapReduce Service
  3. User Guide
  4. MRS Manager Operation Guide
  5. Alarm Reference
  6. ALM-13001 Available ZooKeeper Connections Are Insufficient

ALM-13001 Available ZooKeeper Connections Are Insufficient

Description

The system checks ZooKeeper connections every 30 seconds. This alarm is generated when the system detects that the number of used ZooKeeper instance connections exceeds the threshold (80% of the maximum connections).

This alarm is cleared when the number of used ZooKeeper instance connections is less than the threshold.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

13001

Major

Yes

Parameters

Parameter

Description

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Trigger Condition

Generates an alarm when the actual indicator value exceeds the specified threshold.

Impact on the System

Available ZooKeeper connections are insufficient. When the connection usage reaches 100%, external connections cannot be handled.

Possible Causes

  • The number of connections to the ZooKeeper node exceeds the threshold.
  • Connection leakage occurs on some connection processes.
  • The maximum number of connections does not meet the requirement of the actual scenario.

Procedure

  1. Check the connection status.

    1. On the MRS Manager portal, choose Alarm > ALM-13001 Available ZooKeeper Connections Are Insufficient > Location. Check the IP address of the alarm node.
    2. Obtain the PID of the ZooKeeper process. Log in to the alarm node and run the pgrep -f proc_zookeeper command.
    3. Check whether the PID can be successfully obtained.
    4. Obtain all the IP addresses connected to the ZooKeeper instance and the number of connections. Check 10 IP addresses with the top connections. Run the following command based on the obtained PID and IP address: lsof -i|grep $pid | awk '{print $9}' | cut -d : -f 2 | cut -d \> -f 2 | awk '{a[$1]++} END {for(i in a){print i,a[i] | "sort -r -g -k 2"}}' | head -10. ($pid is the PID obtained in the preceding step.)
    5. Check whether the node IP addresses and the number of connections are successfully obtained.
    6. Obtain the ID of the port connected to the process. Run the following command based on the obtained PID and IP address: lsof -i|grep $pid | awk '{print $9}' |cut -d \> -f 2 |grep $IP | cut -d :-f 2. ($pid and $IP are the PID and IP address obtained in the preceding step.)
    7. Check whether the port ID is successfully obtained.
    8. Obtain the ID of the connected process. Log in to each IP address and run the following command based on the obtained port ID: lsof -i|grep $port. ($port is the port ID obtained in the preceding step.)
    9. Check whether the process ID is successfully obtained.
    10. Check whether connection leakage occurs on the process based on the obtained process ID.
    11. Close the process where connection leakage occurs and check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to Step 2.
    12. On the MRS Manager portal, choose Service > ZooKeeper > Service Configuration > All > quorumpeer > Performance and change the value of maxCnxns to 20000 or more.
    13. Check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to Step 2.

  2. Collect fault information.

    1. On MRS Manager, choose System > Export Log.
    2. Contact technical support engineers for help, detail see technical support.

Related Information

N/A