ALM-45001 Faulty HetuEngine Compute Instances

This alarm applies only to MRS 3.2.0-LTS.2 or later.

Description

The system checks the HetuEngine compute instance status every 60 seconds. This alarm is generated when a HetuEngine compute instance is faulty.

This alarm is cleared when all faulty HetuEngine compute instances are restored.

Attribute

Alarm ID

Alarm Severity

Auto Clear

45001

Major

Yes

Parameters

Name

Meaning

Source

Specifies the cluster for which the alarm is generated.

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Impact on the System

HetuEngine tasks fail to execute.

Possible Causes

  • The HDFS service is abnormal.

  • The Yarn service is abnormal.

  • Yarn queue resources are insufficient.

  • The process of compute instances is faulty.

Procedure

Check the HDFS service status.

  1. In the alarm list, check whether the "ALM-14000 HDFS Service Unavailable" alarm is generated.

    • If yes, go to 2.

    • If no, go to 4.

  2. Clear "ALM-14000 HDFS Service Unavailable" according to the alarm help.

  3. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.

    • If yes, no further action is required.

    • If no, go to 4.

Check the YARN service status.

  1. In the alarm list, check whether the "ALM-18000 YARN Service Unavailable" alarm is generated.

    • If yes, go to 5.

    • If no, go to 7.

  2. Clear "ALM-18000 YARN Service Unavailable" according to the alarm help.

  3. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.

    • If yes, no further action is required.

    • If no, go to 7.

Check the YARN queue resource status.

  1. In the alarm list, check whether the "ALM-18022 Insufficient YARN Queue Resources" alarm is generated.

    • If yes, go to 8.

    • If no, go to 10.

  2. Clear "ALM-18022 Insufficient YARN Queue Resources" according to the alarm help.

  3. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.

    • If yes, no further action is required.

    • If no, go to 10.

Check the HetuEngine compute instance status.

  1. Log in to FusionInsight Manager as an administrator who can access the HetuEngine web UI and choose Cluster > Services > HetuEngine.

  2. In the Basic Information area on the Dashboard tab page, click the link next to HSConsole WebUI to access the HSConsole page.

  3. On the compute instance page, check whether any compute instances are in the FAULT state.

    • If yes, go to 13.

    • If no, go to 14.

  4. In the Operation column of the target compute instance, click Start and wait until the instance is started.

  5. In the alarm list, check whether the "ALM-45001 Faulty HetuEngine Compute Instances" alarm is cleared.

    • If yes, no further action is required.

    • If no, go to 15.

Collect fault information.

  1. On FusionInsight Manager, choose O&M > Log > Download.

  2. Expand the Service drop-down list. In the Services dialog box that is displayed, select HetuEngine under the target cluster name, and click OK.

  3. Expand the Hosts drop-down list. In the Select Host dialog box that is displayed, select the hosts to which the role belongs, and click OK.

  4. Click image1 in the upper right corner, and set Start Date and End Date for log collection to 30 minutes ahead of and after the alarm generation time respectively. Then, click Download.

  5. Contact O&M personnel and provide the collected logs.

Alarm Clearing

This alarm is automatically cleared after the fault is rectified.

last updated: 2024-04-27 01:13