ALM-26051 Storm Service Unavailable

Description

The system checks the Storm service status every 30 seconds. This alarm is generated when all Nimbus nodes in the cluster are abnormal and the Storm service is unavailable.

This alarm is cleared when the Storm service recovers.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

26051

Critical

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Impact on the System

The cluster cannot provide the Storm service, and users cannot perform new Storm tasks.

Possible Causes

  • The Kerberos cluster is faulty.

  • The ZooKeeper cluster is faulty or suspended.

  • The active and standby Nimbus nodes in the Storm cluster are abnormal

Procedure

Check the status of the Kerberos cluster. (Skip this step if the normal mode is used.)

  1. On the FusionInsight Manager portal, choose Cluster > Name of the desired cluster > Services.

  2. Check whether the running status of the Kerberos service is Normal.

    • If yes, go to 5.

    • If no, go to 3.

  3. See the related maintenance information of ALM-25500 KrbServer Service Unavailable.

  4. Check whether the alarm is cleared.

    • If yes, no further action is required.

    • If no, go to 5.

Check the status of the ZooKeeper cluster.

  1. Check whether the running status of the ZooKeeper service is Normal.

    • If yes, go to 8.

    • If no, go to 6.

  2. If ZooKeeper service is stopped, start it, else see the related maintenance information of ALM-13000 ZooKeeper Service Unavailable.

  3. Check whether the alarm is cleared.

    • If yes, no further action is required.

    • If no, go to 8.

Check the status of the active and standby Nimbus nodes.

  1. Choose Cluster > Name of the desired cluster > Services > Storm > Nimbus to go to the Nimbus Instances page.

  2. Check whether only one Nimbus node that is in the Active state in Roles.

    • If yes, go to 13.

    • If no, go to 10.

  3. Select two Nimbus role instances, choose More > Restart Instance, and check whether the instances restart successfully.

    • If yes, go to 11.

    • If no, go to 13.

  4. Log in to the FusionInsight Manager portal again, choose Cluster > Name of the desired cluster > Services > Storm > Nimbus to check whether the running status is Normal.

    • If yes, go to 12.

    • If no, go to 13.

  5. Wait for 30 seconds and check whether the alarm is cleared.

    • If yes, no further action is required.

    • If no, go to 13.

Collecting Fault Information

  1. On the FusionInsight Manager, choose O&M > Log > Download.

  2. Select the following nodes in the required cluster from the Service drop-down list:

    • KrbServer

      Note

      KrbServer logs do not need to be downloaded in normal mode.

    • ZooKeeper

    • Storm

  3. Click image1 in the upper right corner, and set Start Date and End Date for log collection to 10 minutes ahead of and after the alarm generation time, respectively. Then, click Download.

  4. Contact the O&M personnel and send the collected logs.

Alarm Clearing

After the fault is rectified, the system automatically clears this alarm.

last updated: 2024-05-08 03:04