Storm Log Introduction¶
This section applies to MRS 3.x or later.
Log Description¶
Log paths: The default paths of Storm log files are /var/log/Bigdata/storm/Role name (run logs) and /var/log/Bigdata/audit/storm/Role name (audit logs).
Nimbus: /var/log/Bigdata/storm/nimbus (run logs) and /var/log/Bigdata/audit/storm/nimbus (audit logs)
Supervisor: /var/log/Bigdata/storm/supervisor (run logs) and /var/log/Bigdata/audit/storm/supervisor (audit logs)
UI: /var/log/Bigdata/storm/ui (run logs) and /var/log/Bigdata/audit/storm/ui (audit logs)
Logviewer: /var/log/Bigdata/storm/logviewer (run logs) and /var/log/Bigdata/audit/storm/logviewer (audit logs)
Log archive rule: The automatic Storm log compression function is enabled. By default, when the size of logs exceeds 10 MB, logs are automatically compressed into a log file named in the following format: <Original log name>.log.[ID].gz. A maximum of 20 latest compressed files are reserved by default. You can configure the number of compressed files and the compression threshold.
Names of compressed audit log files are in the format of audit.log.[yyyy-MM-dd].[ID].zip. These files permanently exist.
Log Type | Log File Name | Description |
|---|---|---|
Run log | nimbus/access.log | Nimbus user access log |
nimbus/nimbus-<PID>-gc.log | GC log of the Nimbus process | |
nimbus/checkavailable.log | Nimbus availability check log | |
nimbus/checkService.log | Nimbus serviceability check log | |
nimbus/metrics.log | Nimbus monitoring statistics log | |
nimbus/nimbus.log | Run log of the Nimbus process | |
nimbus/postinstall.log | Work log after Nimbus installation | |
nimbus/prestart.log | Work log before Nimbus startup | |
nimbus/start.log | Work log of Nimbus startup | |
nimbus/stop.log | Work log of Nimbus shutdown | |
supervisor/access.log | Supervisor access log | |
supervisor/metrics.log | Supervisor monitoring statistics log | |
supervisor/postinstall.log | Work log after supervisor installation | |
supervisor/prestart.log | Work log before supervisor startup | |
supervisor/start.log | Work log of supervisor startup | |
supervisor/stop.log | Work log of supervisor shutdown | |
supervisor/supervisor.log | Run log of the supervisor process | |
supervisor/supervisor-<PID>-gc.log | GC log of the supervisor process | |
ui/access.log | UI access log | |
ui/metric.log | UI monitoring statistics log | |
ui/ui-<PID>-gc.log | GC log of the UI process | |
ui/postinstall.log | Work log after UI installation | |
ui/prestart.log | Work log before UI startup | |
ui/start.log | Work log of UI startup | |
ui/stop.log | Work log of UI shutdown | |
ui/ui.log | Run log of the UI process | |
logviewer/access.log | Logviewer access log | |
logviewer/metric.log | Logviewer monitoring statistics log | |
logviewer/logviewer-<PID>-gc.log | GC log file of the logviewer process | |
logviewer/logviewer.log | Run log of the logviewer process | |
logviewer/postinstall.log | Work log after logviewer installation | |
logviewer/prestart.log | Work log before logviewer startup | |
logviewer/start.log | Work log of logviewer startup | |
logviewer/stop.log | Work log of logviewer shutdown | |
supervisor/[topologyId]-worker-[Port number].log | Run log of the Worker process. One port occupies one log file. By default, the system contains five ports: 29100, 29101, 29102, 29103 and 29304. | |
supervisor/metadata/[topologyid]-worker-[Port number].yaml | Worker log metadata file, which is used by logviewer to delete logs. This file is automatically deleted by the logviewer log deletion thread based on certain conditions. | |
nimbus/cleanup.log | Cleanup log of Nimbus uninstallation | |
logviewer/cleanup.log | Cleanup log of logviewer uninstallation | |
ui/cleanup.log | Cleanup log of UI uninstallation | |
supervisor/cleanup.log | Cleanup log of supervisor uninstallation | |
leader_switch.log | Run log file that records the Storm active/standby switchover | |
Audit log | nimbus/audit.log | Nimbus audit log |
ui/audit.log | UI audit log | |
supervisor/audit.log | Supervisor audit log | |
logviewer/audit | Logviewer audit log |
Log Levels¶
Table 2 describes the log levels supported by Storm.
Levels of run logs and audit logs are ERROR, WARN, INFO, and DEBUG from the highest to the lowest priority. Run logs of equal or higher levels are recorded. The higher the specified log level, the fewer the logs recorded.
Level | Description |
|---|---|
ERROR | Logs of this level record error information about system running. |
WARN | Logs of this level record exception information about the current event processing. |
INFO | Logs of this level record normal running status information about the system and events. |
DEBUG | Logs of this level record the system information and system debugging information. |
To modify log levels, perform the following operations:
Go to the All Configurations page of Storm by referring to Modifying Cluster Service Configuration Parameters.
On the menu bar on the left, select the log menu of the target role.
Select a desired log level.
Save the configuration. In the displayed dialog box, click OK to make the configurations take effect.
Log Format¶
The following table lists the Storm log formats:
Log Type | Format | Example |
|---|---|---|
Run log | %d{yyyy-MM-dd HH:mm:ss,SSS} | %-5p | [%t] | %m | %logger (%F:%L) %n | 2015-03-11 23:04:00,241 | INFO | [RMI TCP Connection(2646)-10.0.0.2] | The baseSleepTimeMs [1000] the maxSleepTimeMs [1000] the maxRetries [1] | backtype.storm.utils.StormBoundedExponentialBackoffRetry (StormBoundedExponentialBackoffRetry.java:46) |
<yyyy-MM-dd HH:mm:ss,SSS><HostName><RoleName><logLevel><Message> | 2017-03-28 02:57:52 493 10-5-146-1 storm- INFO Nimbus start normally | |
Audit log | <Username><User IP address><Time><Operation><Operation object><Operation result> | UserName=storm/hadoop, UserIP=10.10.0.2, Time=Tue Mar 10 01:15:35 CST 2015, Operation=Kill, Resource=test, Result=Success |