Introduction to Kafka Logs¶
Log Description¶
Log paths: The default storage path of Kafka logs is /var/log/Bigdata/kafka. The default storage path of audit logs is /var/log/Bigdata/audit/kafka.
Broker: /var/log/Bigdata/kafka/broker (run logs)
Kafka UI: /var/log/Bigdata/kafka/ui (run logs)
Log archive rule: The automatic Kafka log compression function is enabled. By default, when the size of logs exceeds 30 MB, logs are automatically compressed into a log file named in the following format: <Original log file name>-<yyyy-mm-dd_hh-mm-ss>.[ID].log.zip. A maximum of 20 latest compressed files are retained by default. You can configure the number of compressed files and the compression threshold.
Type | Log File Name | Description |
|---|---|---|
Run log | server.log | Server run log of the broker process |
controller.log | Controller run log of the broker process | |
kafka-request.log | Request run log of the broker process | |
log-cleaner.log | Cleaner run log of the broker process | |
state-change.log | State-change run log of the broker process | |
kafkaServer-<SSH_USER>-<DATE>-<PID>-gc.log | GC log of the broker process | |
postinstall.log | Work log after broker installation | |
prestart.log | Work log before broker startup | |
checkService.log | Log that records whether broker starts successfully | |
start.log | Startup log of the broker process | |
stop.log | Stop log of the broker process | |
checkavailable.log | Log that records the health check details of the Kafka service | |
checkInstanceHealth.log | Log that records the health check details of broker instances | |
kafka-authorizer.log | Broker authorization log | |
kafka-root.log | Broker basic log | |
cleanup.log | Cleanup log of broker uninstallation | |
metadata-backup-recovery.log | Broker backup and recovery log | |
ranger-kafka-plugin-enable.log | Log that records the Ranger plug-ins enabled by brokers | |
server.out | Broker JVM log | |
audit.log | Authentication log of the Ranger authentication plug-in. This log is archived in the /var/log/Bigdata/audit/kafka directory. |
Type | Log File Name | Description |
|---|---|---|
Run log | kafka-ui.log | Run log of the Kafka UI process |
postinstall.log | Work log after Kafka UI installation | |
cleanup.log | Cleanup log of Kafka UI uninstallation | |
prestart.log | Work log before Kafka UI startup | |
ranger-kafka-plugin-enable.log | Log that records the Ranger plug-ins enabled by Kafka UI | |
start.log | Startup log of the Kafka UI process | |
stop.log | Stop log of the Kafka UI process | |
start.out | Kafka UI process startup information | |
Audit log | audit.log | Audit log of the KafkaUI service |
Authentication log | kafka-authorizer.log | Run log file of the open-source authentication plug-in of Kafka. This log is archived in the /var/log/Bigdata/audit/kafka/kafkaui directory. |
ranger-authorizer.log | Run log of the Ranger authentication plug-in. This log is archived in the /var/log/Bigdata/audit/kafka/kafkaui directory. |
Log Level¶
Table 3 describes the log levels supported by Kafka.
Levels of run logs are ERROR, WARN, INFO, and DEBUG from the highest to the lowest priority. Run logs of equal or higher levels are recorded. The higher the specified log level, the fewer the logs recorded.
Level | Description |
|---|---|
ERROR | Logs of this level record error information about system running. |
WARN | Logs of this level record exception information about the current event processing. |
INFO | Logs of this level record normal running status information about the system and events. |
DEBUG | Logs of this level record the system information and system debugging information. |
To modify log levels, perform the following operations:
Go to the All Configurations page. See Modifying Cluster Service Configuration Parameters.
On the menu bar on the left, select the log menu of the target role.
Select a desired log level.
Save the configuration. In the displayed dialog box, click OK to make the configurations take effect.
Log Format¶
The following table describes the Kafka log format.
Type | Format | Example |
|---|---|---|
Run log | <yyyy-MM-dd HH:mm:ss,SSS>|<Log Level>|<Thread that generates the log>|<Message in the log>|<Full name of the log event invocation class>(<Log file>:<Row>) | 2015-08-08 11:09:53,483 | INFO | [main] | Loading logs. | kafka.log.LogManager (Logging.scala:68) |
<yyyy-MM-dd HH:mm:ss><HostName><Component name><logLevel><Message> | 2015-08-08 11:09:51 10-165-0-83 Kafka INFO Running kafka-start.sh. |