• MapReduce Service

mrs
  1. Help Center
  2. MapReduce Service
  3. User Guide
  4. MRS Manager Operation Guide
  5. Alarm Reference
  6. ALM-13000 ZooKeeper Service Unavailable

ALM-13000 ZooKeeper Service Unavailable

Description

The system checks the ZooKeeper service status every 30 seconds. This alarm is generated when the ZooKeeper service is unavailable and is cleared when the ZooKeeper service recovers.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

13000

Critical

Yes

Parameters

Parameter

Description

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Impact on the System

  • ZooKeeper fails to provide coordination services for upper-layer components.
  • Components dependent on ZooKeeper may not run properly.

Possible Causes

  • A ZooKeeper instance is abnormal.
  • The disk capacity is insufficient.
  • The network is faulty.
  • The DNS is installed on the ZooKeeper node.

Procedure

Check the ZooKeeper service instance status.

  1. On MRS Manager, choose Service > ZooKeeper > quorumpeer.
  2. Check whether the ZooKeeper instances are normal.

  3. Select instances whose status is not good and choose More > Restart Instance.
  4. Check whether the instance status is good after restart.

  5. On the Alarm tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to Step 6.

    Check the disk status.

  6. On MRS Manager, choose Service > ZooKeeper > quorumpeer, and check the host information of the ZooKeeper instance on each node.
  7. On MRS Manager, click Host.
  8. In the Disk Usage column, check whether the disk space of each node that contains ZooKeeper instances is insufficient (where disk usage exceeds 80%).

  9. Expand disk capacity. For details, see ALM-12017 Insufficient Disk Capacity.
  10. On the Alarm tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to Step 11.

    Check the network status.

  11. On the Linux node that contains the ZooKeeper instance, run the ping command to check whether the host names of other nodes that contain the ZooKeeper instance can be pinged successfully.

  12. Modify the IP addresses in /etc/hosts and add the host name and IP address mapping.
  13. Run the ping command again to check whether the host names of other nodes that contain the ZooKeeper instance can be pinged successfully.

  14. On the Alarm tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to Step 15.

    Check the DNS.

  15. Check whether the DNS is installed on the node that contains the ZooKeeper instance. On the Linux node that contains the ZooKeeper instance, run the cat /etc/resolv.conf command to check whether the file is empty.

  16. Run the service named status command to check whether the DNS is started.

  17. Run the service named stop command to stop the DNS service. If "Shutting down name server BIND waiting for named to shut down (28s)" is displayed, the DNS service is stopped successfully. Comment out any content in /etc/resolv.conf.
  18. On the Alarm tab, check whether the alarm is cleared.

    • If yes, no further action is required.
    • If no, go to Step 19.

  19. Collect fault information.

    1. On MRS Manager, choose System > Export Log.
    2. Contact technical support engineers for help, detail see technical support.

Related Information

N/A