Configuring Non-encrypted Transmission¶
Scenario¶
This section describes how to configure Flume server and client parameters after the cluster and the Flume service are installed to ensure proper running of the service.
Note
By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see Configuring the Encrypted Transmission.
Prerequisites¶
The cluster and Flume service have been installed.
The network environment of the cluster is secure.
Procedure¶
Configure the client parameters of the Flume role.
Use the Flume configuration tool on FusionInsight Manager to configure the Flume role client parameters and generate a configuration file.
Log in to FusionInsight Manager. Choose Cluster > Services > Flume > Configuration Tool.
Set Agent Name to client. Select and drag the source, channel, and sink to be used to the GUI on the right, and connect them.
For example, use SpoolDir Source, File Channel, and Avro Sink, as shown in Figure 1.
Figure 1 Example for the Flume configuration tool¶
Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to Table 1 based on the actual environment.
Note
If the client parameters of the Flume role have been configured, you can obtain the existing client parameter configuration file from client installation directory/fusioninsight-flume-1.9.0/conf/properties.properties to ensure that the configuration is in concordance with the previous. Log in to FusionInsight Manager, choose Cluster > Services > Flume > Configuration > Import, import the file, and modify the configuration items related to non-encrypted transmission.
It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
Click Export to save the properties.properties configuration file to the local server.
Table 1 Parameters to be modified of the Flume role client¶ Parameter
Description
Example Value
ssl
Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.)
Only Sources of the Avro type have this configuration item.
true indicates that the function is enabled.
false indicates that the function is not enabled.
false
Upload the properties.properties file to flume/conf/ under the installation directory of the Flume client.
Configure the server parameters of the Flume role and upload the configuration file to the cluster.
Use the Flume configuration tool on the FusionInsight Manager portal to configure the server parameters and generate the configuration file.
Log in to FusionInsight Manager. Choose Cluster > Services > Flume > Configuration Tool.
Set Agent Name to server. Select and drag the source, channel, and sink to be used to the GUI on the right, and connect them.
For example, use Avro Source, File Channel, and HDFS Sink, as shown in Figure 2.
Figure 2 Example for the Flume configuration tool¶
Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to Table 2 based on the actual environment.
Note
If the server parameters of the Flume role have been configured, you can choose Cluster > Services > Flume > Instance on FusionInsight Manager. Then select the corresponding Flume role instance and click the Download button behind the flume.config.file parameter on the Instance Configurations page to obtain the existing server parameter configuration file. Choose Cluster > Service > Flume > Configurations > Import, import the file, and modify the configuration items related to non-encrypted transmission.
It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
A unique checkpoint directory needs to be configured for each File Channel.
Click Export to save the properties.properties configuration file to the local server.
Table 2 Parameters to be modified of the Flume role server¶ Parameter
Description
Example Value
ssl
Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.)
Only Sources of the Avro type have this configuration item.
true indicates that the function is enabled.
false indicates that the function is not enabled.
false
Log in to FusionInsight Manager and choose Cluster > Services > Flume. On the Instances tab page, click Flume.
Select the Flume role of the node where the configuration file is to be uploaded, choose Instance Configurations > Import beside the flume.config.file, and select the properties.properties file.
Note
An independent server configuration file can be uploaded to each Flume instance.
This step is required for updating the configuration file. Modifying the configuration file on the background is an improper operation because the modification will be overwritten after configuration synchronization.
Click Save, and then click OK.
Click Finish.