• MapReduce Service

mrs
  1. Help Center
  2. MapReduce Service
  3. User Guide
  4. MRS Manager Operation Guide
  5. Alarm Reference
  6. ALM-14011 HDFS DataNode Data Directory Is Not Configured Properly

ALM-14011 HDFS DataNode Data Directory Is Not Configured Properly

Description

The DataNode parameter dfs.datanode.data.dir specifies DataNode data directories. This alarm is generated in any of the following scenarios:

  • A configured data directory cannot be created.
  • A data directory uses the same disk as other critical directories in the system.
  • Multiple directories use the same disk.

This alarm is cleared when the DataNode data directory is configured properly and this DataNode is restarted.

Attribute

Alarm ID

Alarm Severity

Automatically Cleared

14011

Major

Yes

Parameters

Parameter

Description

ServiceName

Specifies the service for which the alarm is generated.

RoleName

Specifies the role for which the alarm is generated.

HostName

Specifies the host for which the alarm is generated.

Impact on the System

If the DataNode data directory is mounted on critical directories such as the root directory, the disk space of the root directory will be used up after running for a long time. This causes a system fault.

If the DataNode data directory is not configured properly, HDFS performance will deteriorate.

Possible Causes

  • The DataNode data directory fails to be created.
  • The DataNode data directory uses the same disk as critical directories, such as / or /boot.
  • Multiple directories in the DataNode data directory use the same disk.

Procedure

  1. Check the alarm cause and information about the DataNode that generated the alarm.

    1. On the MRS Manager portal, click Alarm. In the alarm list, click the alarm.
    2. In the Alarm Details area, view Alarm Cause. In HostName of Location, obtain the host name of the DataNode that generated the alarm.

  2. Delete directories that do not comply with the disk plan from the DataNode data directory.

    1. Choose Service > HDFS > Instance. In the instance list, click the DataNode instance on the alarm node.
    2. Click Instance Configuration and view the value of the DataNode parameter dfs.datanode.data.dir.
    3. Check whether all DataNode data directories are consistent with the disk plan.
      • If yes, go to 2.d.
      • If no, go to 2.g.
    4. Modify the DataNode parameter dfs.datanode.data.dir and delete the incorrect directories.
    5. Choose Service > HDFS > Instance and restart the DataNode instance.
    6. Check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to 2.g.
    7. Log in to the DataNode that generated the alarm.
      • If the alarm cause is "The DataNode data directory fails to be created", go to 3.a.
      • If the alarm cause is "The DataNode data directory uses the same disk as critical directories, such / or /boot", go to 4.a.
      • If the alarm cause is "Multiple directories in the DataNode data directory use the same disk", go to 5.a.

  3. Check whether the DataNode data directory is created.

    1. Run the following command to switch the user:

      sudo su - root

      su - omm

    2. Run the ls command to check whether the directories exist in the DataNode data directory.
    3. Run the mkdir data directory command to create a directory. Check whether the directory is successfully created.
      • If yes, go to 6.a.
      • If no, go to 3.d.
    1. On the MRS Manager portal, click Alarm to check whether alarm ALM-12017 Insufficient Disk Capacity exists.
      • If yes, go to 3.e.
      • If no, go to 3.f.
    2. Adjust the disk capacity and check whether alarm ALM-12017 Insufficient Disk Capacity is cleared. For details, see ALM-12017 Insufficient Disk Capacity.
    3. Check whether user omm has the rwx or x permission for all upper-layer directories of the directory. For example, for /tmp/abc/, user omm has the x permission for the tmp directory and the rwx permission for the abc directory.
      • If yes, go to 6.a.
      • If no, go to 3.g.
    4. Run the chmod u+rwx path or chmod u+x path command as user root to assign the rwx or x permission to user omm. Go to 3.c.

  4. Check whether the DataNode data directory uses the same disk as other critical directories in the system.

    1. Run the df command to obtain the disk mounting information of each directory in the DataNode data directory.
    2. Check whether the directories mounted on the disk are critical directories, such as / or /boot.
      • If yes, go to 4.c.
      • If no, go to 6.a.
    3. Change the value of the DataNode parameter dfs.datanode.data.dir and delete the directories that use the same disk as critical directories.
    4. Go to 6.a.

  5. Check whether multiple directories in the DataNode data directory use the same disk.

    1. Run the df command to obtain the disk mounting information of each directory in the DataNode data directory. Record the mounted directory in the command output.
    2. Modify the DataNode parameter dfs.datanode.data.dir to reserve one of the directories mounted on the same disk directory.
    3. Go to 6.a.

  6. Restart the DataNode and check whether the alarm is cleared.

    1. On the MRS Manager portal, choose Service > HDFS > Instance and restart the DataNode instance.
    2. Check whether the alarm is cleared.
      • If yes, no further action is required.
      • If no, go to Step 7.

  7. Collect fault information.

    1. On MRS Manager, choose System > Export Log.
    2. Contact technical support engineers for help, detail see technical support.

Related Information

N/A