Configuring Container Log Aggregation

Scenario

Yarn provides the container log aggregation function to collect logs generated by containers on each node to HDFS to release local disk space. You can collect logs in either of the following ways:

  • After the application is complete, collect container logs to HDFS at a time.

  • During application running, periodically collect log segments generated by containers and save them to HDFS.

Configuration Description

Navigation path for setting parameters:

Go to the All Configurations page of Yarn and enter a parameter name list in Table 1 in the search box by referring to Modifying Cluster Service Configuration Parameters.

The yarn.nodemanager.remote-app-log-dir-suffix parameter must be configured on the Yarn client. The configurations on the ResourceManager, NodeManager, and JobHistory nodes must be the same as those on the Yarn client.

The periodic log collection function applies only to MapReduce applications, for which rolling output of log files must be configured. Table 3 describes the configurations in the mapred-site.xml configuration file on the MapReduce client node.

Table 1 Parameter description

Parameter

Description

Default Value

yarn.log-aggregation-enable

Whether to enable container log aggregation

  • If this parameter is set to true, logs are collected to the HDFS directory.

  • If this parameter is set to false, the function is disabled, and logs are not collected to HDFS.

After changing the parameter value, restart the Yarn service for the setting to take effect.

Note

  • The container logs that are generated before the parameter is set to false and the setting takes effect cannot be obtained from the web UI.

  • If you want to view the logs generated before on the web UI, you are advised to set this parameter to true.

true

yarn.log-aggregation.per-day.enabled

Whether to enable the function of collecting Yarn job logs by day. After this function is enabled, the logs generated on the same day are stored in a separate directory. This parameter takes effect only when:

  • The value of yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds is not greater than 0.

  • The value of yarn.log-aggregation.retain-seconds is greater than 86400 seconds.

Note

This section applies to MRS 3.2.0 or later.

After this function is enabled, the default log archive directory (configurable) changes to /tmp/logs/{USER_NAME}/{DATE_STR}/logs/application_id.

  • USER_NAME: indicates the user who runs the application.

  • DATE_STR: indicates the time when the application is complete.

For example, if user testuser runs a job whose ID is application_11111111_1111 and the job is completed on December 12, 2021, the job log will be stored on the /tmp/logs/testuser/20211212/logs/application_11111111_1111 directory.

false

yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds

Interval for NodeManager to periodically collect logs

  • If this parameter is set to -1 or 0, periodic log collection is disabled. Logs are collected at a time after application running is complete.

  • The minimum collection interval can be set to 3,600 seconds. If this parameter is set to a value greater than 0 and less than 3,600, the collection interval is 3,600 seconds.

Interval for NodeManager to wake up and upload logs. If this parameter is set to -1 or 0, rolling monitoring is disabled and logs are aggregated when the application task is complete. The value must be greater than or equal to -1.

-1

yarn.nodemanager.disk-health-checker.log-dirs.max-disk-utilization-per-disk-percentage

Maximum percentage of the Yarn disk quota that can be occupied by the container log directory on each disk. When the space occupied by the log directory exceeds the value of this parameter, the periodic log collection service is triggered to start a log collection activity beyond the period to release the local disk space. Maximum space for container logs that can be provided on each disk. If the disk space occupied by container logs exceeds this threshold, data aggregation in rolling mode is triggered.

The valid value range of the maximum disk quota percentage is -1 to 100. If the value is less than or equal to -1, it is forcibly reset to 25. If the value is greater than 100, the value is forcibly reset to 25. If you set the value to a negative number, the disk capacity detection function for Container log directory is disabled.

Note

  • Percentage of the available disk space of the container log directory = Percentage of the available disk space of Yarn (yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage) x Percentage of the available disk space of the container log directory (yarn.nodemanager.disk-health-checker.log-dirs.max-disk-utilization-per-disk-percentage)

  • Only applications with the periodic log collection function enabled can trigger log collection when the disk quota of the log directory exceeds the threshold.

25

yarn.nodemanager.remote-app-log-dir-suffix

Name of the HDFS folder in which container logs are to be stored. This parameter and yarn.nodemanager.remote-app-log-dir form the full path for storing container logs. That is, {yarn.nodemanager.remote-app-log-dir}/${user}/{yarn.nodemanager.remote-app-log-dir-suffix}.

Note

{user} indicates the username for running the task.

logs

yarn.nodemanager.log-aggregator.on-fail.remain-log-in-sec

Duration for retaining container logs on the local host after the logs fail to be collected, in second

  • If this parameter is set to a negative number or 0, local logs are deleted immediately.

  • If this parameter is set to a positive number, local logs are retained for this period.

    Note

    This parameter takes effect only when yarn.nodemanager.delete.debug-delay-sec is not set to -1. The default value is 0. yarn.nodemanager.delete.debug-delay-sec also controls the file deletion function. If this parameter is set to -1, other functions will be changed. Exercise caution when setting this parameter.

604800

Go to the All Configurations page of MapReduce and enter a parameter name in Table 2 in the search box by referring to Modifying Cluster Service Configuration Parameters.

Table 2 Parameter description

Parameter

Description

Default Value

yarn.log-aggregation.retain-seconds

Duration for retaining aggregated logs, in second

  • If this parameter is set to a negative integer, the container logs will be retained permanently in the HDFS.

  • If this parameter is set to 0 or a positive integer, container logs will be stored for such a period and deleted after the period expires.

    Note

    A short period may increase load of the NameNode. Therefore, you are advised to set this parameter to a proper value.

1296000

yarn.log-aggregation.retain-check-interval-seconds

Interval for storing container logs in HDFS, in second

  • If this parameter is set to -1 or 0, the interval will be one tenth of the period specified by yarn.log-aggregation.retain-seconds.

    Note

    If this parameter is set to -1 or 0, yarn.log-aggregation.retain-seconds cannot be set to 0.

  • If this parameter is set to a positive number, container logs in HDFS will be scanned at such an interval.

    Note

    A short interval may increase load of the NameNode. Therefore, you are advised to set this parameter to a proper value.

86400

Go to the All Configurations page of Yarn and enter a parameter name list in Table 3 in the search box by referring to Modifying Cluster Service Configuration Parameters.

Table 3 Configuring rolling output of MapReduce application log files

Parameter

Description

Default Value

mapreduce.task.userlog.limit.kb

Maximum size of a single task log file of the MapReduce application. When the maximum size of the log file has been reached, a new log file is generated. The value 0 indicates that the size of the log file is not limited.

51200

yarn.app.mapreduce.task.container.log.backups

Maximum number of task logs that can be retained for the MapReduce application.

If this parameter is set to 0, rolling output is disabled.

Number of task log backup files when ContainerRollingLogAppender (CRLA) is used. By default, ContainerLogAppender (CLA) is used and container logs are not rolled back.

When both mapreduce.task.userlog.limit.kb and yarn.app.mapreduce.task.container.log.backups are greater than 0, CRLA is enabled. The value ranges from 0 to 999.

10

yarn.app.mapreduce.am.container.log.limit.kb

Maximum size of a single ApplicationMaster log file of the MapReduce application, in KB. When the maximum size of the log file has been reached, a new log file is generated. The value 0 indicates that the size of a single ApplicationMaster log file is not limited.

51200

yarn.app.mapreduce.am.container.log.backups

Maximum number of ApplicationMaster logs that can be retained for the MapReduce application. If this parameter is set to 0, rolling output is disabled. Number of ApplicationMaster log backup files when CRLA is used. By default, CLA is used and container logs are not rolled back.

When both yarn.app.mapreduce.am.container.log.limit.kb and yarn.app.mapreduce.am.container.log.backups are greater than 0, CRLA is enabled for the ApplicationMaster. The value ranges from 0 to 999.

20

yarn.app.mapreduce.shuffle.log.backups

Maximum number of shuffle logs that can be retained for the MapReduce application. If this parameter is set to 0, rolling output is disabled.

When both yarn.app.mapreduce.shuffle.log.limit.kb and yarn.app.mapreduce.shuffle.log.backups are greater than 0, syslog.shuffle uses CRLA. The value ranges from 0 to 999.

10

yarn.app.mapreduce.shuffle.log.limit.kb

Maximum size of a single shuffle log file of the MapReduce application, in KB. When the maximum size of the log file has been reached, a new log file is generated. If this parameter is set to 0, the size of a single shuffle log file is not limited. The value must be greater than or equal to 0.

51200

last updated: 2024-11-14 12:28 UTC - commit: 9d95be538b931742820cff066f12ce18f037ed96