Flume Service Configuration Guide¶
This configuration guide describes how to configure common Flume services. For non-common Source, Channel, and Sink configuration, see the user manual provided by the Flume community.
Note
Parameters in bold in the following tables are mandatory.
The value of BatchSize of the Sink must be less than that of transactionCapacity of the Channel.
Only some parameters of Source, Channel, and Sink are displayed on the Flume configuration tool page. For details, see the following configurations.
The Customer Source, Customer Channel, and Customer Sink displayed on the Flume configuration tool page need to be configured based on self-developed code. The following common configurations are not displayed.
Common Source Configurations¶
Avro Source
An Avro source listens to the Avro port, receives data from the external Avro client, and places data into configured channels. Common configurations are as follows:
Table 1 Common configurations of an Avro source¶ Parameter
Default Value
Description
channels
-Specifies the channel connected to the source. Multiple channels can be configured.
type
avro
Specifies the type of the avro source, which must be avro.
bind
-Specifies the listening host name/IP address.
port
-Specifies the bound listening port. Ensure that this port is not occupied.
threads
-Specifies the maximum number of source threads.
compression-type
none
Specifies the message compression format, which can be set to none or deflate. none indicates that data is not compressed, while deflate indicates that data is compressed.
compression-level
6
Specifies the data compression level, which ranges from 1 to 9. The larger the value is, the higher the compression rate is.
ssl
false
Specifies whether to use SSL encryption. If this parameter is set to true, the values of keystore and keystore-password must be specified.
truststore-type
JKS
Specifies the Java trust store type, which can be set to JKS or PKCS12.
Note
Different passwords are used to protect the key store and private key of JKS, while the same password is used to protect the key store and private key of PKCS12.
truststore
-Specifies the Java trust store file.
truststore-password
-Specifies the Java trust store password.
keystore-type
JKS
Specifies the keystore type set after SSL is enabled, which can be set to JKS or PKCS12.
Note
Different passwords are used to protect the key store and private key of JKS, while the same password is used to protect the key store and private key of PKCS12.
keystore
-Specifies the keystore file path set after SSL is enabled. This parameter is mandatory if SSL is enabled.
keystore-password
-Specifies the keystore password set after SSL is enabled. This parameter is mandatory if SSL is enabled.
trust-all-certs
false
Specifies whether to disable the check for the SSL server certificate. If this parameter is set to true, the SSL server certificate of the remote source is not checked. You are not advised to perform this operation during the production.
exclude-protocols
SSLv3
Specifies the excluded protocols. The entered protocols must be separated by spaces. The default value is SSLv3.
ipFilter
false
Specifies whether to enable the IP address filtering.
ipFilter.rules
-Specifies the rules of N network ipFilters. Host names or IP addresses must be separated by commas (,). If this parameter is set to true, there are two configuration rules: allow and forbidden. The configuration format is as follows:
ipFilterRules=allow:ip:127.*, allow:name:localhost, deny:ip:*
SpoolDir Source
SpoolDir Source monitors and transmits new files that have been added to directories in real-time mode. Common configurations are as follows:
Table 2 Common configurations of a Spooling Directory source¶ Parameter
Default Value
Description
channels
-Specifies the channel connected to the source. Multiple channels can be configured.
type
spooldir
Specifies the type of the spooling source, which must be set to spooldir.
spoolDir
-Specifies the monitoring directory of the Spooldir source. A Flume running user must have the read, write, and execution permissions on the directory.
monTime
0 (Disabled)
Specifies the thread monitoring threshold. When the update time exceeds the threshold, the source is restarted. Unit: second
fileSuffix
.COMPLETED
Specifies the suffix added after file transmission is complete.
deletePolicy
never
Specifies the source file deletion policy after file transmission is complete. The value can be either never or immediate. never indicates that the source file is not deleted after file transmission is complete, while immediate indicates that the source file is immediately deleted after file transmission is complete.
ignorePattern
^$
Specifies the regular expression of a file to be ignored. The default value is ^$, indicating that spaces are ignored.
includePattern
^.*$
Specifies the regular expression that contains a file. This parameter can be used together with ignorePattern. If a file meets both ignorePattern and includePattern, the file is ignored. In addition, when a file starts with a period (.), the file will not be filtered.
trackerDir
.flumespool
Specifies the metadata storage path during data transmission.
batchSize
1000
Specifies the number of events written to the channel in batches.
decodeErrorPolicy
FAIL
Specifies the code error policy.
Note
If a code error occurs in the file, set decodeErrorPolicy to REPLACE or IGNORE. Flume will skip the code error and continue to collect subsequent logs.
deserializer
LINE
Specifies the file parser. The value can be either LINE or BufferedLine.
When the value is set to LINE, characters read from the file are transcoded one by one.
When the value is set to BufferedLine, one line or multiple lines of characters read from the file are transcoded in batches, which delivers better performance.
deserializer.maxLineLength
2048
Specifies the maximum length for resolution by line.
deserializer.maxBatchLine
1
Specifies the maximum number of lines for resolution by line. If multiple lines are set, maxLineLength must be set to a corresponding multiplier.
Note
When configuring the Interceptor, take the multi-line combination into consideration to avoid data loss. If the Interceptor cannot process combined lines, set this parameter to 1.
selector.type
replicating
Specifies the selector type. The value can be either replicating or multiplexing. replicating indicates that data is replicated and then transferred to each channel so that each channel receives the same data, while multiplexing indicates that a channel is selected based on the value of the header in the event and each channel has different data.
interceptors
-Specifies the interceptor. Multiple interceptors are separated by spaces.
inputCharset
UTF-8
Specifies the encoding format of a read file. The encoding format must be the same as that of the data source file that has been read. Otherwise, an error may occur during character parsing.
fileHeader
false
Specifies whether to add the file name (including the file path) to the event header.
fileHeaderKey
-Specifies that the data storage structure in header is set in the <key,value> mode. Parameters fileHeaderKey and fileHeader must be used together. Following is an example if fileHeader is set to true:
Define fileHeaderKey as file. When the /root/a.txt file is read, fileHeaderKey exists in the header in the file=/root/a.txt format.
basenameHeader
false
Specifies whether to add the file name (excluding the file path) to the event header.
basenameHeaderKey
-Specifies that the data storage structure in header is set in the <key,value> mode. Parameters basenameHeaderKey and basenameHeader must be used together. Following is an example if basenameHeader is set to true:
Define basenameHeaderKey as file. When the a.txt file is read, basenameHeaderKey exists in the header in the file=a.txt format.
pollDelay
500
Specifies the delay for polling new files in the monitoring directory. Unit: milliseconds
recursiveDirectorySearch
false
Specifies whether to monitor new files in the subdirectory of the configured directory.
consumeOrder
oldest
Specifies the consumption order of files in a directory. If this parameter is set to oldest or youngest, the sequence of files to be read is determined by the last modification time of files in the monitored directory. If there are a large number of files in the directory, it takes a long time to search for oldest or youngest files. If this parameter is set to random, an earlier created file may not be read for a long time. If this parameter is set to oldest or youngest, it takes a long time to find the latest and the earliest file. The options are as follows: random, youngest, and oldest.
maxBackoff
4000
Specifies the maximum time to wait between consecutive attempts to write to a channel if the channel is full. If the time exceeds the threshold, an exception is thrown. The corresponding source starts to write at a smaller time value. Each time the source attempts, the digital exponent increases until the current specified value is reached. If data cannot be written, the data write fails. Unit: second
emptyFileEvent
true
Specifies whether to collect empty file information and send it to the sink end. The default value is true, indicating that empty file information is sent to the sink end. This parameter is valid only for HDFS Sink. Taking HDFS Sink as an example, if this parameter is set to true and an empty file exists in the spoolDir directory, an empty file with the same name will be created in the hdfs.path directory of HDFS.
Note
SpoolDir Source ignores the last line feed character of each event when data is reading by row. Therefore, Flume does not calculate the data volume counters used by the last line feed character.
Kafka Source
A Kafka source consumes data from Kafka topics. Multiple sources can consume data of the same topic, and the sources consume different partitions of the topic. Common configurations are as follows:
Table 3 Common configurations of a Kafka source¶ Parameter
Default Value
Description
channels
-Specifies the channel connected to the source. Multiple channels can be configured.
type
org.apache.flume.source.kafka.KafkaSource
Specifies the type of the Kafka source, which must be set to org.apache.flume.source.kafka.KafkaSource.
kafka.bootstrap.servers
-Specifies the bootstrap address port list of Kafka. If Kafka has been installed in the cluster and the configuration has been synchronized to the server, you do not need to set this parameter on the server. The default value is the list of all brokers in the Kafka cluster. This parameter must be configured on the client. Use commas (,) to separate multiple values of IP address:Port number. The rules for matching ports and security protocols must be as follows: port 21007 matches the security mode (SASL_PLAINTEXT), and port 9092 matches the common mode (PLAINTEXT).
kafka.topics
-Specifies the list of subscribed Kafka topics, which are separated by commas (,).
kafka.topics.regex
-Specifies the subscribed topics that comply with regular expressions. kafka.topics.regex has a higher priority than kafka.topics and will overwrite kafka.topics.
monTime
0 (Disabled)
Specifies the thread monitoring threshold. When the update time exceeds the threshold, the source is restarted. Unit: second
nodatatime
0 (Disabled)
Specifies the alarm threshold. An alarm is triggered when the duration that Kafka does not release data to subscribers exceeds the threshold. Unit: second This parameter can be configured in the properties.properties file.
batchSize
1000
Specifies the number of events written to the channel in batches.
batchDurationMillis
1000
Specifies the maximum duration of topic data consumption at a time, expressed in milliseconds.
keepTopicInHeader
false
Specifies whether to save topics in the event header. If the parameter value is true, topics configured in Kafka Sink become invalid.
setTopicHeader
true
If this parameter is set to true, the topic name defined in topicHeader is stored in the header.
topicHeader
topic
When setTopicHeader is set to true, this parameter specifies the name of the topic received by the storage device. If the property is used with that of Kafka Sink topicHeader, be careful not to send messages to the same topic cyclically.
useFlumeEventFormat
false
By default, an event is transferred from a Kafka topic to the body of the event in the form of bytes. If this parameter is set to true, the Avro binary format of Flume is used to read events. When used together with the parseAsFlumeEvent parameter with the same name in KafkaSink or KakfaChannel, any set header generated from the data source is retained.
keepPartitionInHeader
false
Specifies whether to save partition IDs in the event header. If the parameter value is true, Kafka Sink writes data to the corresponding partition.
kafka.consumer.group.id
flume
Specifies the Kafka consumer group ID. Sources or proxies having the same ID are in the same consumer group.
kafka.security.protocol
SASL_PLAINTEXT
Specifies the Kafka security protocol. The parameter value must be set to PLAINTEXT in a common cluster. The rules for matching ports and security protocols must be as follows: port 21007 matches the security mode (SASL_PLAINTEXT), and port 9092 matches the common mode (PLAINTEXT).
Other Kafka Consumer Properties
-Specifies other Kafka configurations. This parameter can be set to any consumption configuration supported by Kafka, and the .kafka prefix must be added to the configuration.
Taildir Source
A Taildir source monitors file changes in a directory and automatically reads the file content. In addition, it can transmit data in real time. Common configurations are as follows:
Table 4 Common configurations of a Taildir source¶ Parameter
Default Value
Description
channels
-Specifies the channel connected to the source. Multiple channels can be configured.
type
TAILDIR
Specifies the type of the taildir source, which must be set to TAILDIR.
filegroups
-Specifies the group name of a collection file directory. Group names are separated by spaces.
filegroups.<filegroupName>.parentDir
-Specifies the parent directory. The value must be an absolute path.
filegroups.<filegroupName>.filePattern
-Specifies the relative file path of the file group's parent directory. Directories can be included and regular expressions are supported. It must be used together with parentDir.
positionFile
-Specifies the metadata storage path during data transmission.
headers.<filegroupName>.<headerKey>
-Specifies the key-value of an event when data of a group is being collected.
byteOffsetHeader
false
Specifies whether each event header contains the event location information in the source file. If the parameter value is true, the location information is saved in the byteoffset variable.
maxBatchCount
Long.MAX_VALUE
Specifies the maximum number of batches that can be consecutively read from a file. If the monitored directory reads multiple files consecutively and one of the files is written at a rapid rate, other files may fail to be processed. This is because the file that is written at a high speed will be in an infinite read loop. In this case, set this parameter to a smaller value.
skipToEnd
false
Specifies whether Flume can locate the latest location of a file and read the latest data after restart. If the parameter value is true, Flume locates and reads the latest file data after restart.
idleTimeout
120000
Specifies the idle duration during file reading, expressed in milliseconds. If file content is not changed in the preset time duration, close the file. If data is written to this file after the file is closed, open the file and read data.
writePosInterval
3000
Specifies the interval for writing metadata to a file, expressed in milliseconds.
batchSize
1000
Specifies the number of events written to the channel in batches.
monTime
0 (Disabled)
Specifies the thread monitoring threshold. When the update time exceeds the threshold, the source is restarted. Unit: second
fileHeader
false
Specifies whether to add the file name (including the file path) to the event header.
fileHeaderKey
file
Specifies that the data storage structure in header is set in the <key,value> mode. Parameters fileHeaderKey and fileHeader must be used together. Following is an example if fileHeader is set to true:
Define fileHeaderKey as file. When the /root/a.txt file is read, fileHeaderKey exists in the header in the file=/root/a.txt format.
Http Source
An HTTP source receives data from an external HTTP client and sends the data to the configured channels. Common configurations are as follows:
Table 5 Common configurations of an HTTP source¶ Parameter
Default Value
Description
channels
-Specifies the channel connected to the source. Multiple channels can be configured.
type
http
Specifies the type of the http source, which must be set to http.
bind
-Specifies the listening host name/IP address.
port
-Specifies the bound listening port. Ensure that this port is not occupied.
handler
org.apache.flume.source.http.JSONHandler
Specifies the message parsing method of an HTTP request. Two formats are supported: JSON (org.apache.flume.source.http.JSONHandler) and BLOB (org.apache.flume.sink.solr.morphline.BlobHandler).
handler.*
-Specifies handler parameters.
exclude-protocols
SSLv3
Specifies the excluded protocols. The entered protocols must be separated by spaces. The default value is SSLv3.
include-cipher-suites
-Specifies the included protocols. The entered protocols must be separated by spaces. If this parameter is left empty, all protocols are supported by default.
enableSSL
false
Specifies whether SSL is enabled in HTTP. If this parameter is set to true, the values of keystore and keystore-password must be specified.
keystore-type
JKS
Specifies the keystore type, which can be JKS or PKCS12.
keystore
-Specifies the keystore path set after SSL is enabled in HTTP.
keystorePassword
-Specifies the keystore password set after SSL is enabled in HTTP.
Thrift Source
Thrift Source monitors the thrift port, receives data from the external Thrift clients, and puts the data into the configured channel. Common configurations are as follows:
Parameter
Default Value
Description
channels
-Specifies the channel connected to the source. Multiple channels can be configured.
type
thrift
Specifies the type of the thrift source, which must be set to thrift.
bind
-Specifies the listening host name/IP address.
port
-Specifies the bound listening port. Ensure that this port is not occupied.
threads
-Specifies the maximum number of worker threads that can be run.
kerberos
false
Specifies whether Kerberos authentication is enabled.
agent-keytab
-Specifies the address of the keytab file used by the server. The machine-machine account must be used. You are advised to use flume/conf/flume_server.keytab in the Flume service installation directory.
agent-principal
-Specifies the principal of the security user used by the server. The principal must be a machine-machine account. You are advised to use the default user of Flume: flume_server/hadoop.<system domain name>@<system domain name>
Note
flume_server/hadoop.<system domain name> is the username. All letters in the system domain name contained in the username are lowercase letters. For example, Local Domain is set to 9427068F-6EFA-4833-B43E-60CB641E5B6C.COM, and the username is flume_server/hadoop.9427068f-6efa-4833-b43e-60cb641e5b6c.com.
compression-type
none
Specifies the message compression format, which can be set to none or deflate. none indicates that data is not compressed, while deflate indicates that data is compressed.
ssl
false
Specifies whether to use SSL encryption. If this parameter is set to true, the values of keystore and keystore-password must be specified.
keystore-type
JKS
Specifies the keystore type set after SSL is enabled.
keystore
-Specifies the keystore file path set after SSL is enabled. This parameter is mandatory if SSL is enabled.
keystore-password
-Specifies the keystore password set after SSL is enabled. This parameter is mandatory if SSL is enabled.
Common Channel Configurations¶
Memory Channel
A memory channel uses memory as the cache. Events are stored in memory queues. Common configurations are as follows:
Table 6 Common configurations of a memory channel¶ Parameter
Default Value
Description
type
-Specifies the type of the memory channel, which must be set to memory.
capacity
10000
Specifies the maximum number of events cached in a channel.
transactionCapacity
1000
Specifies the maximum number of events accessed each time.
Note
The parameter value must be greater than the batchSize of the source and sink.
The value of transactionCapacity must be less than or equal to that of capacity.
channelfullcount
10
Specifies the channel full count. When the count reaches the threshold, an alarm is reported.
keep-alive
3
Specifies the waiting time of the Put and Take threads when the transaction or channel cache is full. Unit: second
byteCapacity
80% of the maximum JVM memory
Specifies the total bytes of all event bodies in a channel. The default value is the 80% of the maximum JVM memory (indicated by -Xmx). Unit: bytes
byteCapacityBufferPercentage
20
Specifies the percentage of bytes in a channel (%).
File Channel
A file channel uses local disks as the cache. Events are stored in the folder specified by dataDirs. Common configurations are as follows:
Table 7 Common configurations of a file channel¶ Parameter
Default Value
Description
type
-Specifies the type of the file channel, which must be set to file.
checkpointDir
${BIGDATA_DATA_HOME}/hadoop/data1~N/flume/checkpoint
Note
This path is changed with the custom data path.
Specifies the checkpoint storage directory.
dataDirs
${BIGDATA_DATA_HOME}/hadoop/data1~N/flume/data
Note
This path is changed with the custom data path.
Specifies the data cache directory. Multiple directories can be configured to improve performance. The directories are separated by commas (,).
maxFileSize
2146435071
Specifies the maximum size of a single cache file, expressed in bytes.
minimumRequiredSpace
524288000
Specifies the minimum idle space in the cache, expressed in bytes.
capacity
1000000
Specifies the maximum number of events cached in a channel.
transactionCapacity
10000
Specifies the maximum number of events accessed each time.
Note
The parameter value must be greater than the batchSize of the source and sink.
The value of transactionCapacity must be less than or equal to that of capacity.
channelfullcount
10
Specifies the channel full count. When the count reaches the threshold, an alarm is reported.
useDualCheckpoints
false
Specifies the backup checkpoint. If this parameter is set to true, the backupCheckpointDir parameter value must be set.
backupCheckpointDir
-Specifies the path of the backup checkpoint.
checkpointInterval
30000
Specifies the check interval, expressed in seconds.
keep-alive
3
Specifies the waiting time of the Put and Take threads when the transaction or channel cache is full. Unit: second
use-log-replay-v1
false
Specifies whether to enable the old reply logic.
use-fast-replay
false
Specifies whether to enable the queue reply.
checkpointOnClose
true
Specifies that whether a checkpoint is created when a channel is disabled.
Memory File Channel
A memory file channel uses both memory and local disks as its cache and supports message persistence. It provides similar performance as a memory channel and better performance than a file channel. This channel is currently experimental and not recommended for use in production. The following table describes common configuration items: Common configurations are as follows:
Table 8 Common configurations of a memory file channel¶ Parameter
Default Value
Description
type
org.apache.flume.channel.MemoryFileChannel
Specifies the type of the memory file channel, which must be set to org.apache.flume.channel.MemoryFileChannel.
capacity
50000
Specifies the maximum number of events cached in a channel.
transactionCapacity
5000
Specifies the maximum number of events processed by a transaction.
Note
The parameter value must be greater than the batchSize of the source and sink.
The value of transactionCapacity must be less than or equal to that of capacity.
subqueueByteCapacity
20971520
Specifies the maximum size of events that can be stored in a subqueue, expressed in bytes.
A memory file channel uses both queues and subqueues to cache data. Events are stored in a subqueue, and subqueues are stored in a queue.
subqueueCapacity and subqueueInterval determine the size of events that can be stored in a subqueue. subqueueCapacity specifies the capacity of a subqueue, and subqueueInterval specifies the duration that a subqueue can store events. Events in a subqueue are sent to the destination only after the subqueue reaches the upper limit of subqueueCapacity or subqueueInterval.
Note
The value of subqueueByteCapacity must be greater than the number of events specified by batchSize.
subqueueInterval
2000
Specifies the maximum duration that a subqueue can store events, expressed in milliseconds.
keep-alive
3
Specifies the waiting time of the Put and Take threads when the transaction or channel cache is full.
Unit: second
dataDir
-Specifies the cache directory for local files.
byteCapacity
80% of the maximum JVM memory
Specifies the channel cache capacity.
Unit: bytes
compression-type
None
Specifies the message compression format, which can be set to none or deflate. none indicates that data is not compressed, while deflate indicates that data is compressed.
channelfullcount
10
Specifies the channel full count. When the count reaches the threshold, an alarm is reported.
The following is a configuration example of a memory file channel:
server.channels.c1.type = org.apache.flume.channel.MemoryFileChannel server.channels.c1.dataDir = /opt/flume/mfdata server.channels.c1.subqueueByteCapacity = 20971520 server.channels.c1.subqueueInterval=2000 server.channels.c1.capacity = 500000 server.channels.c1.transactionCapacity = 40000
Kafka Channel
A Kafka channel uses a Kafka cluster as the cache. Kafka provides high availability and multiple copies to prevent data from being immediately consumed by sinks when Flume or Kafka Broker crashes.
Table 9 Common configurations of a Kafka channel¶ Parameter
Default Value
Description
type
-Specifies the type of the Kafka channel, which must be set to org.apache.flume.channel.kafka.KafkaChannel.
kafka.bootstrap.servers
-Specifies the bootstrap address port list of Kafka.
If Kafka has been installed in the cluster and the configuration has been synchronized to the server, you do not need to set this parameter on the server. The default value is the list of all brokers in the Kafka cluster. This parameter must be configured on the client. Use commas (,) to separate multiple values of IP address:Port number. The rules for matching ports and security protocols must be as follows: port 21007 matches the security mode (SASL_PLAINTEXT), and port 9092 matches the common mode (PLAINTEXT).
kafka.topic
flume-channel
Specifies the Kafka topic used by the channel to cache data.
kafka.consumer.group.id
flume
Specifies the data group ID obtained from Kafka. This parameter cannot be left blank.
parseAsFlumeEvent
true
Specifies whether data is parsed into Flume events.
migrateZookeeperOffsets
true
Specifies whether to search for offsets in ZooKeeper and submit them to Kafka when there is no offset in Kafka.
kafka.consumer.auto.offset.reset
latest
Specifies where to consume if there is no offset record, which can be set to earliest, latest, or none. earliest indicates that the offset is reset to the initial point, latest indicates that the offset is set to the latest position, and none indicates that an exception is thrown if there is no offset.
kafka.producer.security.protocol
SASL_PLAINTEXT
Specifies the Kafka producer security protocol. The rules for matching ports and security protocols must be as follows: port 21007 matches the security mode (SASL_PLAINTEXT), and port 9092 matches the common mode (PLAINTEXT).
Note
If the parameter is not displayed, click + in the lower left corner of the dialog box to display all parameters.
kafka.consumer.security.protocol
SASL_PLAINTEXT
Specifies the Kafka consumer security protocol. The rules for matching ports and security protocols must be as follows: port 21007 matches the security mode (SASL_PLAINTEXT), and port 9092 matches the common mode (PLAINTEXT).
pollTimeout
500
Specifies the maximum timeout interval for the consumer to invoke the poll function. Unit: milliseconds
ignoreLongMessage
false
Specifies whether to discard oversized messages.
messageMaxLength
1000012
Specifies the maximum length of a message written by Flume to Kafka.
Common Sink Configurations¶
HDFS Sink
An HDFS sink writes data into HDFS. Common configurations are as follows:
Table 10 Common configurations of an HDFS sink¶ Parameter
Default Value
Description
channel
-Specifies the channel connected to the sink.
type
hdfs
Specifies the type of the hdfs sink, which must be set to hdfs.
hdfs.path
-Specifies the data storage path in HDFS. The value must start with hdfs://hacluster/.
monTime
0 (Disabled)
Specifies the thread monitoring threshold. When the update time exceeds the threshold, the sink is restarted. Unit: second
hdfs.inUseSuffix
.tmp
Specifies the suffix of the HDFS file to which data is being written.
hdfs.rollInterval
30
Specifies the interval for file rolling, expressed in seconds.
hdfs.rollSize
1024
Specifies the size for file rolling, expressed in bytes.
hdfs.rollCount
10
Specifies the number of events for file rolling.
Note
Parameters rollInterval, rollSize, and rollCount can be configured at the same time. The parameter meeting the requirements takes precedence for compression.
hdfs.idleTimeout
0
Specifies the timeout interval for closing idle files automatically, expressed in seconds.
hdfs.batchSize
1000
Specifies the number of events written into HDFS in batches.
hdfs.kerberosPrincipal
-Specifies the Kerberos principal of HDFS authentication. This parameter is mandatory in a secure mode, but not required in a common mode.
hdfs.kerberosKeytab
-Specifies the Kerberos keytab of HDFS authentication. This parameter is not required in a common mode, but in a secure mode, the Flume running user must have the permission to access keyTab path in the jaas.cof file.
hdfs.fileCloseByEndEvent
true
Specifies whether to close the HDFS file when the last event of the source file is received.
hdfs.batchCallTimeout
-Specifies the timeout control duration when events are written into HDFS in batches. Unit: milliseconds
If this parameter is not specified, the timeout duration is controlled when each event is written into HDFS. When the value of hdfs.batchSize is greater than 0, configure this parameter to improve the performance of writing data into HDFS.
Note
The value of hdfs.batchCallTimeout depends on hdfs.batchSize. A greater hdfs.batchSize requires a larger hdfs.batchCallTimeout. If the value of hdfs.batchCallTimeout is too small, writing events to HDFS may fail.
serializer.appendNewline
true
Specifies whether to add a line feed character (\n) after an event is written to HDFS. If a line feed character is added, the data volume counters used by the line feed character will not be calculated by HDFS sinks.
hdfs.filePrefix
over_%{basename}
Specifies the file name prefix after data is written to HDFS.
hdfs.fileSuffix
-Specifies the file name suffix after data is written to HDFS.
hdfs.inUsePrefix
-Specifies the prefix of the HDFS file to which data is being written.
hdfs.fileType
DataStream
Specifies the HDFS file format, which can be set to SequenceFile, DataStream, or CompressedStream.
Note
If the parameter is set to SequenceFile or DataStream, output files are not compressed, and the codeC parameter cannot be configured. However, if the parameter is set to CompressedStream, the output files are compressed, and the codeC parameter must be configured together.
hdfs.codeC
-Specifies the file compression format, which can be set to gzip, bzip2, lzo, lzop, or snappy.
hdfs.maxOpenFiles
5000
Specifies the maximum number of HDFS files that can be opened. If the number of opened files reaches this value, the earliest opened files are closed.
hdfs.writeFormat
Writable
Specifies the file write format, which can be set to Writable or Text.
hdfs.callTimeout
10000
Specifies the timeout control duration each time events are written into HDFS, expressed in milliseconds.
hdfs.threadsPoolSize
-Specifies the number of threads used by each HDFS sink for HDFS I/O operations.
hdfs.rollTimerPoolSize
-Specifies the number of threads used by each HDFS sink to schedule the scheduled file rolling.
hdfs.round
false
Specifies whether to round off the timestamp value. If this parameter is set to true, all time-based escape sequences (except %t) are affected.
hdfs.roundUnit
second
Specifies the unit of the timestamp value that has been rounded off, which can be set to second, minute, or hour.
hdfs.useLocalTimeStamp
true
Specifies whether to enable the local timestamp. The recommended parameter value is true.
hdfs.closeTries
0
Specifies the maximum attempts for the hdfs sink to stop renaming a file. If the parameter is set to the default value 0, the sink does not stop renaming the file until the file is successfully renamed.
hdfs.retryInterval
180
Specifies the interval of request for closing the HDFS file, expressed in seconds.
Note
For each closing request, there are multiple RPCs working on the NameNode back and forth, which may make the NameNode overloaded if the parameter value is too small. Also, when the parameter is set to 0, the Sink will not attempt to close the file, but opens the file or uses .tmp as the file name extension, if the first closing attempt fails.
hdfs.failcount
10
Specifies the number of times that data fails to be written to HDFS. If the number of times that the sink fails to write data to HDFS exceeds the parameter value, an alarm indicating abnormal data transmission is reported.
Avro Sink
An Avro sink converts events into Avro events and sends them to the monitoring ports of the hosts. Common configurations are as follows:
Table 11 Common configurations of an Avro sink¶ Parameter
Default Value
Description
channel
-Specifies the channel connected to the sink.
type
-Specifies the type of the avro sink, which must be set to avro.
hostname
-Specifies the bound host name or IP address.
port
-Specifies the bound listening port. Ensure that this port is not occupied.
batch-size
1000
Specifies the number of events sent in a batch.
client.type
DEFAULT
Specifies the client instance type. Set this parameter based on the communication protocol used by the configured model. The options are as follows:
DEFAULT: The client instance of the AvroRPC type is returned.
OTHER: NULL is returned.
THRIFT: The client instance of the Thrift RPC type is returned.
DEFAULT_LOADBALANCING: The client instance of the LoadBalancing RPC type is returned.
DEFAULT_FAILOVER: The client instance of the Failover RPC type is returned.
ssl
false
Specifies whether to use SSL encryption. If this parameter is set to true, the values of keystore and keystore-password must be specified.
truststore-type
JKS
Specifies the Java trust store type, which can be set to JKS or PKCS12.
Note
Different passwords are used to protect the key store and private key of JKS, while the same password is used to protect the key store and private key of PKCS12.
truststore
-Specifies the Java trust store file.
truststore-password
-Specifies the Java trust store password.
keystore-type
JKS
Specifies the keystore type set after SSL is enabled.
keystore
-Specifies the keystore file path set after SSL is enabled. This parameter is mandatory if SSL is enabled.
keystore-password
-Specifies the keystore password after SSL is enabled. This parameter is mandatory if SSL is enabled.
connect-timeout
20000
Specifies the timeout for the first connection, expressed in milliseconds.
request-timeout
20000
Specifies the maximum timeout for a request after the first request, expressed in milliseconds.
reset-connection-interval
0
Specifies the interval between a connection failure and a second connection, expressed in seconds. If the parameter is set to 0, the system continuously attempts to perform a connection.
compression-type
none
Specifies the compression type of the batch data, which can be set to none or deflate. none indicates that data is not compressed, while deflate indicates that data is compressed. This parameter value must be the same as that of the AvroSource compression-type.
compression-level
6
Specifies the compression level of batch data, which can be set to 1 to 9. A larger value indicates a higher compression rate.
exclude-protocols
SSLv3
Specifies the excluded protocols. The entered protocols must be separated by spaces. The default value is SSLv3.
HBase Sink
An HBase sink writes data into HBase. Common configurations are as follows:
Table 12 Common configurations of an HBase sink¶ Parameter
Default Value
Description
channel
-Specifies the channel connected to the sink.
type
-Specifies the type of the HBase sink, which must be set to hbase.
table
-Specifies the HBase table name.
columnFamily
-Specifies the HBase column family.
monTime
0 (Disabled)
Specifies the thread monitoring threshold. When the update time exceeds the threshold, the sink is restarted. Unit: second
batchSize
1000
Specifies the number of events written into HBase in batches.
kerberosPrincipal
-Specifies the Kerberos principal of HBase authentication. This parameter is mandatory in a secure mode, but not required in a common mode.
kerberosKeytab
-Specifies the Kerberos keytab of HBase authentication. This parameter is not required in a common mode, but in a secure mode, the Flume running user must have the permission to access keyTab path in the jaas.cof file.
coalesceIncrements
true
Specifies whether to perform multiple operations on the same hbase cell in a same processing batch. Setting this parameter to true improves performance.
Kafka Sink
A Kafka sink writes data into Kafka. Common configurations are as follows:
Table 13 Common configurations of a Kafka sink¶ Parameter
Default Value
Description
channel
-Specifies the channel connected to the sink.
type
-Specifies the type of the kafka sink, which must be set to org.apache.flume.sink.kafka.KafkaSink.
kafka.bootstrap.servers
-Specifies the bootstrap address port list of Kafka. If Kafka has been installed in the cluster and the configuration has been synchronized to the server, you do not need to set this parameter on the server. The default value is the list of all brokers in the Kafka cluster. The client must be configured with this parameter. If there are multiple values, use commas (,) to separate the values. The rules for matching ports and security protocols must be as follows: port 21007 matches the security mode (SASL_PLAINTEXT), and port 9092 matches the common mode (PLAINTEXT).
monTime
0 (Disabled)
Specifies the thread monitoring threshold. When the update time exceeds the threshold, the sink is restarted. Unit: second
kafka.producer.acks
1
Successful write is determined by the number of received acknowledgement messages about replicas. The value 0 indicates that no confirm message needs to be received, the value 1 indicates that the system is only waiting for only the acknowledgement information from a leader, and the value -1 indicates that the system is waiting for the acknowledgement messages of all replicas. If this parameter is set to -1, data loss can be avoided in some leader failure scenarios.
kafka.topic
-Specifies the topic to which data is written. This parameter is mandatory.
flumeBatchSize
1000
Specifies the number of events written into Kafka in batches.
kafka.security.protocol
SASL_PLAINTEXT
Specifies the Kafka security protocol. The parameter value must be set to PLAINTEXT in a common cluster. The rules for matching ports and security protocols must be as follows: port 21007 matches the security mode (SASL_PLAINTEXT), and port 9092 matches the common mode (PLAINTEXT).
ignoreLongMessage
false
Specifies whether to discard oversized messages.
messageMaxLength
1000012
Specifies the maximum length of a message written by Flume to Kafka.
defaultPartitionId
-Specifies the Kafka partition ID to which the events of a channel is transferred. The partitionIdHeader value overwrites this parameter value. By default, if this parameter is left blank, events will be distributed by the Kafka Producer's partitioner (by a specified key or a partitioner customized by kafka.partitioner.class).
partitionIdHeader
-When you set this parameter, the sink will take the value of the field named using the value of this property from the event header and send the message to the specified partition of the topic. If the value does not have a valid partition, EventDeliveryException is thrown. If the header value already exists, this setting overwrites the defaultPartitionId parameter.
Other Kafka Producer Properties
-Specifies other Kafka configurations. This parameter can be set to any production configuration supported by Kafka, and the .kafka prefix must be added to the configuration.
Thrift Sink
A Thrift sink converts events to Thrift events and sends them to the monitoring port of the configured host. Common configurations are as follows:
Table 14 Common configurations of a Thrift sink¶ Parameter
Default Value
Description
channel
-Specifies the channel connected to the sink.
type
thrift
Specifies the type of the thrift sink, which must be set to thrift.
hostname
-Specifies the bound host name or IP address.
port
-Specifies the bound listening port. Ensure that this port is not occupied.
batch-size
1000
Specifies the number of events sent in a batch.
connect-timeout
20000
Specifies the timeout for the first connection, expressed in milliseconds.
request-timeout
20000
Specifies the maximum timeout for a request after the first request, expressed in milliseconds.
kerberos
false
Specifies whether Kerberos authentication is enabled.
client-keytab
-Specifies the path of the client keytab file. The Flume running user must have the access permission on the authentication file.
client-principal
-Specifies the principal of the security user used by the client.
server-principal
-Specifies the principal of the security user used by the server.
compression-type
none
Specifies the compression type of data sent by Flume, which can be set to none or deflate. none indicates that data is not compressed, while deflate indicates that data is compressed.
maxConnections
5
Specifies the maximum size of the connection pool for Flume to send data.
ssl
false
Specifies whether to use SSL encryption.
truststore-type
JKS
Specifies the Java trust store type.
truststore
-Specifies the Java trust store file.
truststore-password
-Specifies the Java trust store password.
reset-connection-interval
0
Specifies the interval between a connection failure and a second connection, expressed in seconds. If the parameter is set to 0, the system continuously attempts to perform a connection.
Precautions¶
What are the reliability measures of Flume?
Use the transaction mechanisms between Source and Channel as well as between Channel and Sink.
Configure the failover and load_balance mechanisms for Sink Processor. The following shows a load balancing example.
server.sinkgroups=g1 server.sinkgroups.g1.sinks=k1 k2 server.sinkgroups.g1.processor.type=load_balance server.sinkgroups.g1.processor.backoff=true server.sinkgroups.g1.processor.selector=random
What are the precautions for the aggregation and cascading of multiple Flume agents?
Avro or Thrift protocol can be used for cascading.
When the aggregation end contains multiple nodes, evenly distribute the agents and do not aggregate all agents on a single node.