Configuring HDFS DiskBalancer¶
Scenario¶
DiskBalancer is an online disk balancer that balances disk data on running DataNodes based on various indicators. It works in the similar way of the HDFS Balancer. The difference is that HDFS Balancer balances data between DataNodes, while HDFS DiskBalancer balances data among disks on a single DataNode.
Data among disks may be unevenly distributed if a large number of files have been deleted from a cluster running for a long time, or disk capacity expansion is performed on a node in the cluster. Uneven data distribution may deteriorate the concurrent read/write performance of the HDFS, or cause service failure due to inappropriate HDFS write policies. In this case, the data density among disks on a node needs to be balanced to prevent heterogeneous small disks from becoming the performance bottleneck of the node.
Configuration Description¶
Go to the All Configurations page of HDFS and enter a parameter name in the search box by referring to Modifying Cluster Service Configuration Parameters.
Parameter | Description | Default Value |
|---|---|---|
dfs.disk.balancer.auto.enabled | Indicates whether to enable the HDFS DiskBalancer function. The default value is false, indicating that this function is disabled. | false |
dfs.disk.balancer.auto.cron.expression | CRON expression of the HDFS disk balancing operation, which is used to control the start time of the balancing operation. This parameter is valid only when dfs.disk.balancer.auto.enabled is set to true. The default value is 0 1 * * 6, indicating that tasks are executed at 01:00 every Saturday. For details about cron expression, see Table 2. The default value indicates that the DiskBalancer check is executed at 01:00 every Saturday. | 0 1 * * 6 |
dfs.disk.balancer.max.disk.throughputInMBperSec | Specifies the maximum disk bandwidth that can be used for disk data balancing. The unit is MB/s, and the default value is 10. Set this parameter based on the actual disk conditions of the cluster. | 10 |
dfs.disk.balancer.max.disk.errors | Specifies the maximum number of errors that are allowed in a specified movement process. If the value exceeds this threshold, the movement fails. | 5 |
dfs.disk.balancer.block.tolerance.percent | Specifies the difference threshold between the data storage capacity and perfect status of each disk during data balancing among disks. For example, the ideal data storage capacity of each disk is 1 TB, and this parameter is set to 10. When the data storage capacity of the target disk reaches 900 GB, the storage status of the disk is considered as perfect. Value range: 1 to 100. | 10 |
dfs.disk.balancer.plan.threshold.percent | Specifies the data density difference that is allowed between two disks during disk data balancing. If the absolute value of the data density difference between any two disks exceeds the threshold, data balancing is required. Value range: 1 to 100. | 10 |
dfs.disk.balancer.top.nodes.number | Specifies the top N nodes whose disk data needs to be balanced in the cluster. | 5 |
To use this function, set dfs.disk.balancer.auto.enabled to true and configure a proper CRON expression. Set other parameters based on the cluster status.
Column | Description |
|---|---|
1 | Minute. The value ranges from 0 to 59. |
2 | Hour. The value ranges from 0 to 23. |
3 | Date. The value ranges from 1 to 31. |
4 | Month. The value ranges from 1 to 12. |
5 | Week. The value ranges from 0 to 6. 0 indicates Sunday. |
Use Restrictions¶
Data can only be moved between disks of the same type. For example, data can only be moved between SSDs or between DISKs.
Enabling this function occupies disk I/O resources and network bandwidth resources of involved nodes. Enable this function in off-peak hours.
The DataNodes specified by the dfs.disk.balancer.top.nodes.number parameter is frequently calculated. Therefore, set the parameter to a small value.
Commands for using the DiskBalancer function on the HDFS client are as follows:
Table 3 DiskBalancer commands¶ Syntax
Description
hdfs diskbalancer -report -top <N>
Set N to an integer greater than 0. This command can be used to query the top N nodes that require disk data balancing in the cluster.
hdfs diskbalancer -plan <Hostname| IP Address>
This command can be used to generate a JSON file based on the DataNode. The file contains information about the source disk, target disk, and blocks to be moved. In addition, this command can be used to specify other parameters such as the network bandwidth.
hdfs diskbalancer -query <Hostname:$dfs.datanode.ipc.port>
The default port number of the cluster is 9867. This command is used to query the running status of the DiskBalancer task on the current node.
hdfs diskbalancer -execute <planfile>
In this command, planfile indicates the JSON file generated in the second command. Use the absolute path.
hdfs diskbalancer -cancel <planfile>
This command is used to cancel the running planfile. Use the absolute path.
Note
Users running this command on the client must have the supergroup permission. You can use the system user hdfs of the HDFS service. Alternatively, you can create a user with the supergroup permission in the cluster and then run the command.
Only formats and usage of commands are provided in Table 3. For more parameters to be configured for each command, run the hdfs diskbalancer -help <command> command to view detailed information.
When you troubleshoot performance problems during the cluster O&M, check whether the HDFS disk balancing occurs in the event information of the cluster. If yes, check whether DiskBalancer is enabled in the cluster.
After the automatic DiskBalancer function is enabled, the ongoing task stops only after the current data balancing is complete. The task cannot be canceled during the balancing.
You can manually specify certain nodes for data balancing on the client.