Yarn Log Overview¶
Log Description¶
The default paths for saving Yarn logs are as follows:
ResourceManager: /var/log/Bigdata/yarn/rm (run logs) and /var/log/Bigdata/audit/yarn/rm (audit logs)
NodeManager: /var/log/Bigdata/yarn/nm (run logs) and /var/log/Bigdata/audit/yarn/nm (audit logs)
Log archive rule: The automatic compression and archive function is enabled for Yarn logs. By default, when the size of a log file exceeds 50 MB, the log file is automatically compressed. The naming rule of the compressed log file is as follows: <Original log file name>-<yyyy-mm-dd_hh-mm-ss>.[ID].log.zip. A maximum of 100 latest compressed files are retained. The number of compressed files can be configured on Manager.
Log archive rule:
Log Type | Log File Name | Description |
|---|---|---|
Run log | hadoop-<SSH_USER>-<process_name>-<hostname>.log | Yarn component log file, which records most of the logs generated when the Yarn component is running |
yarn-<SSH_USER>-<process_name>-<hostname>.out | Log file that records Yarn running environment information | |
<process_name>-<SSH_USER>-<DATE>-<PID>-gc.log | Garbage collection log file | |
yarn-haCheck.log | ResourceManager active/standby status detection log file | |
yarn-service-check.log | Log file that records the health check details of the Yarn service | |
yarn-start-stop.log | Log file that records the startup and stop of the Yarn service | |
yarn-prestart.log | Log file that records cluster operations before the Yarn service startup | |
yarn-postinstall.log | Work log file after installation and before startup of the Yarn service | |
hadoop-commission.log | Yarn service entry log file | |
yarn-cleanup.log | Log file that records the cleanup operation during uninstallation of the Yarn service | |
yarn-refreshqueue.log | Yarn queue refresh log file | |
upgradeDetail.log | Upgrade log file | |
stderr/stdin/syslog | Container log file of the applications running on the Yarn service | |
yarn-application-check.log | Check log file of applications running on the Yarn service | |
yarn-appsummary.log | Running result log file of applications running on the Yarn service | |
yarn-switch-resourcemanager.log | Run log file that records the Yarn active/standby switchover | |
ranger-yarn-plugin-enable.log | Log file that records the enabling of Ranger authentication for Yarn | |
yarn-nodemanager-period-check.log | Periodic check log of Yarn NodeManager | |
yarn-resourcemanager-period-check.log | Periodic check log of Yarn ResourceManager | |
hadoop.log | Hadoop client logs | |
env.log | Environment information log file before the instance is started or stopped. | |
Audit logs | yarn-audit-<process_name>.log ranger-plugin-audit.log | Yarn operation audit log file |
SecurityAuth.audit | Yarn security audit log file |
Log Level¶
Table 2 describes the log levels supported by Yarn, including OFF, FATAL, ERROR, WARN, INFO, and DEBUG, from high priority to low. Logs whose levels are higher than or equal to the specified level are printed. The number of printed logs decreases as the specified log level increases.
Level | Description |
|---|---|
FATAL | Logs of this level record critical error information about the current event processing. |
ERROR | Logs of this level record error information about the current event processing. |
WARN | Logs of this level record exception information about the current event processing. |
INFO | Logs of this level record normal running status information about the system and events. |
DEBUG | Logs of this level record the system as well as system debugging information. |
To modify log levels, perform the following operations:
Go to the All Configurations page of the Yarn service by referring to Modifying Cluster Service Configuration Parameters.
On the menu bar on the left, select the log menu of the target role.
Select a desired log level.
Click Save Configuration. In the dialog box that is displayed, click OK to make the setting take effect.
Note
The configurations take effect immediately without the need to restart the service.
Log Format¶
The following table lists the Yarn log formats.
Log Type | Format | Example |
|---|---|---|
Run log | <yyyy-MM-dd HH:mm:ss,SSS>|<Log Level>|<Thread that generates the log>|<Message in the log>|<Location of the log event> | 2014-09-26 14:18:59,109 | INFO | main | Client environment:java.compiler=<NA> | org.apache.zookeeper.Environment.logEnv(Environment.java:100) |
Audit log | <yyyy-MM-dd HH:mm:ss,SSS>|<Log Level>|<Thread that generates the log>|<Message in the log>|<Location of the log event> | 2014-09-26 14:24:43,605 | INFO | main-EventThread | USER=omm OPERATION=refreshAdminAcls TARGET=AdminService RESULT=SUCCESS | org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger$LogLevel$6.printLog(RMAuditLogger.java:91) |