Installing the Flume Client on Clusters¶
Scenario¶
To use Flume to collect logs, you must install the Flume client on a log host.
Prerequisites¶
A cluster with the Flume component has been created.
The log host is in the same VPC and subnet with the MRS cluster. Log in to the node where the client is installed. For details, seeUsing an MRS Client on Nodes Outside a MRS Cluster.
You have obtained the username and password for logging in to the log host.
The installation directory is automatically created if it does not exist. If it exists, the directory must be left blank. The directory path cannot contain any space.
Procedure¶
Obtain the software package.
Log in to the FusionInsight Manager. Choose Cluster > Name of the target cluster > Services > Flume. On the Flume service page that is displayed, choose More > Download Client in the upper right corner and set Select Client Type to Complete Client to download the Flume service client file.
The file name of the client is FusionInsight_Cluster_<Cluster ID>_Flume_Client.tar. This section takes the client file FusionInsight_Cluster_1_Flume_Client.tar as an example.
Upload the software package.
Upload the software package to a directory, for example, /opt/client on the node where the Flume service client will be installed as user user.
Note
user is the user who installs and runs the Flume client.
Decompress the software package.
Log in to the node where the Flume service client is to be installed as user user. Go to the directory where the installation package is installed, for example, /opt/client, and run the following command to decompress the installation package to the current directory:
cd /opt/client
tar -xvf FusionInsight_Cluster_1_Flume_Client.tar
Verify the software package.
Run the sha256sum -c command to verify the decompressed file. If OK is returned, the verification is successful. Example:
sha256sum -c FusionInsight_Cluster_1_Flume_ClientConfig.tar.sha256
FusionInsight_Cluster_1_Flume_ClientConfig.tar: OK
Decompress the package.
tar -xvf FusionInsight_Cluster_1_Flume_ClientConfig.tar
Check whether the number of clients is 1.
Run the following command in the Flume client installation directory to install the client to a specified directory (for example, opt/FlumeClient): After the client is installed successfully, the installation is complete.
cd /opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/Flume/FlumeClient
./install.sh -d /opt/FlumeClient -f MonitorServerService IP address or host name of the role -c User service configuration filePath for storing properties.properties -s CPU threshold -l /var/log/Bigdata -e FlumeServer service IP address or host name -n Flume
Note
-d: Flume client installation path
(Optional) -f: IP addresses or host names of two MonitorServer roles. The IP addresses or host names are separated by commas (,). If this parameter is not configured, the Flume client does not send alarm information to MonitorServer and information about the client cannot be viewed on the FusionInsight Manager GUI.
(Optional) -c: Service configuration file, which needs to be generated by the user based on the service. For details about how to generate the file on the configuration tool page of the Flume server, see Flume Service Configuration Guide. Upload the file to any directory on the node where the client is to be installed. If this parameter is not specified during the installation, you can upload the generated service configuration file properties.properties to the /opt/FlumeClient/fusioninsight-flume-1.9.0/conf directory after the installation.
(Optional) -s: cgroup threshold. The value is an integer ranging from 1 to 100 x N. N indicates the number of CPU cores. The default threshold is -1, indicating that the processes added to the cgroup are not restricted by the CPU usage.
(Optional) -l: Log path. The default value is /var/log/Bigdata. The user user must have the write permission on the directory. When the client is installed for the first time, a subdirectory named flume-client is generated. After the installation, subdirectories named flume-client-n will be generated in sequence. The letter n indicates a sequence number, which starts from 1 in ascending order. In the /conf/ directory of the Flume client installation directory, modify the ENV_VARS file and search for the FLUME_LOG_DIR attribute to view the client log path.
(Optional) -e: Service IP address or host name of FlumeServer, which is used to receive statistics for the monitoring indicator reported by the client.
(Optional) -n: Name of the Flume client. You can choose Cluster > Name of the desired cluster > Service > Flume > Flume Management on FusionInsight Manager to view the client name on the corresponding node.
If the following error message is displayed, run the export JAVA_HOME=JDK path command.
JAVA_HOME is null in current user,please install the JDK and set the JAVA_HOME
IBM JDK does not support -Xloggc. You must change -Xloggc to -Xverbosegclog in flume/conf/flume-env.sh. For 32-bit JDK, the value of -Xmx must not exceed 3.25 GB.
When installing a cross-platform client in a cluster, go to the /opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/Flume/FusionInsight-Flume-1.9.0.tar.gz directory to install the Flume client.
Go to the directory for installing clients in batches.
cd /opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/Flume/FlumeClient/batch_install
Note
When installing a cross-platform client in a cluster, go to the /opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/Flume/FusionInsight-Flume-1.9.0.tar.gz directory to install the Flume client.
Configure the host_info.cfg file. The format of the configuration file is as follows:
host_ip="",user="",password="",install_path="",flume_config_file="",monitor_server_ip="",log_path="",flume_server_ip="",cgroup_threshold="",client_name=""
Note
(Mandatory) host_ip: IP address of the node where the Flume client is to be installed.
(Mandatory) user: User name for logging in to the node where the Flume client is to be installed remotely.
(Mandatory) password: Password for logging in to the Flume client to be installed remotely.
(Mandatory) install_path: Installation path of the Flume client.
(Optional) flume_config_file: Configuration file for Flume running. You are advised to specify this configuration file during Flume installation. If you do not set this parameter, retain the value "" and do not delete the parameter.
(Optional) monitor_server_ip: Service IP address of the Flume MonitorServer in the cluster. You can check the IP address on FusionInsight Manager. You can select either of the two IP addresses. If the IP address is not configured, the client does not send alarm information to the cluster in the scenario where a process is faulty.
(Optional) log_path: Path for storing Flume run logs. If this parameter is not set, logs are recorded in /var/log/Bigdata/flume-client
-Index by default. Index value: If there is only one client in this path, the value is 1. If there are multiple clients, the index value is incremented by 1.(Optional) flume_server_ip: Service IP address of the Flume server. The indicator information of the client is reported to the cluster from this node. The indicator information about the client can be displayed on the web client. If the indicator information is not configured, the client does not display the indicator information.
(Optional) cgroup_threshold: cgroup threshold. The value is an integer ranging from 1 to 100 x N. N indicates the number of CPU cores. The default threshold is -1, indicating that the processes added to the cgroup are not restricted by the CPU usage.
(Optional) client_name: Client name. The client name is displayed on the client monitoring page. If the client name is not configured, the client name is empty.
Run the following command to install the Flume client in batches.
./batch_install.sh -p /opt/client/FusionInsight_Cluster_1_Flume_Client.tar
Delete the password information from the host_info.cfg file.
After the batch installation is complete, delete the password information from the host_info.cfg file immediately. Otherwise, the password may be disclosed.