ALM-14026 Blocks on DataNode Exceed the Threshold¶
Description¶
The system checks the number of blocks on each DataNode every 30 seconds. This alarm is generated when the number of blocks on the DataNode exceeds the threshold.
If Trigger Count is 1 and the number of blocks on the DataNode is less than or equal to the threshold, this alarm is cleared. If Trigger Count is greater than 1 and the number of blocks on the DataNode is less than or equal to 90% of the threshold, this alarm is cleared.
Attribute¶
Alarm ID | Alarm Severity | Auto Clear |
---|---|---|
14026 | Minor | Yes |
Parameters¶
Name | Meaning |
---|---|
Source | Specifies the cluster for which the alarm is generated. |
ServiceName | Specifies the service for which the alarm is generated. |
RoleName | Specifies the role for which the alarm is generated. |
HostName | Specifies the host for which the alarm is generated. |
Trigger condition | Specifies the threshold for triggering the alarm. |
Impact on the System¶
If this alarm is reported, there are too many blocks on the DataNode. In this case, data writing into the HDFS may fail due to insufficient disk space.
Possible Causes¶
The alarm threshold is improperly configured.
Data skew occurs among DataNodes.
The disk space configured for the HDFS cluster is insufficient.
Procedure¶
Change the threshold.
On FusionInsight Manager, choose Cluster, click the name of the desired cluster, and choose HDFS. Then choose Configurations > All Configurations. On the displayed page, find the GC_OPTS parameter under HDFS->DataNode.
Set the threshold of the DataNode blocks. Specifically, change the value of Xmx of the GC_OPTS parameter. Xmx specifies the memory, and each GB memory supports a maximum of 500,000 DataNode blocks. Set the memory as required. Confirm that GC_PROFILE is set to custom and save the configuration.
Choose Cluster, click the name of the desired cluster, and choose HDFS > Instance. Select the DataNode instance whose status is Expired, click More, and select Restart Instance to make the GC_OPTS configuration take effect.
Check whether the alarm is cleared 5 minutes later.
If yes, no further action is required.
If no, go to 5.
Check whether associated alarms are reported.
On FusionInsight Manager, choose O&M > Alarm > Alarms, and check whether the ALM-14002 DataNode Disk Usage Exceeds the Threshold alarm exists.
Handle the alarm by following the instructions in ALM-14002 DataNode Disk Usage Exceeds the Threshold and check whether the alarm is cleared.
Check whether the alarm is cleared 5 minutes later.
If yes, no further action is required.
If no, go to 8.
Expand the DataNode capacity.
Expand the DataNode capacity.
On FusionInsight Manager, wait for 5 minutes and check whether the alarm is cleared.
If yes, no further action is required.
If no, go to 10.
Collect the fault information.
On FusionInsight Manager, choose O&M. In the navigation pane on the left, choose Log > Download.
Expand the drop-down list next to the Service field. In the Services dialog box that is displayed, select HDFS for the target cluster.
Click in the upper right corner, and set Start Date and End Date for log collection to 20 minutes ahead of and after the alarm generation time, respectively. Then, click Download.
Contact O&M personnel and provide the collected logs.
Alarm Clearing¶
This alarm is automatically cleared after the fault is rectified.