Link to Hive

Description

By creating a Hive link, you can extract data from or load data to Hive of MRS.

Sample Link

The following is the message body of a sample link. You are advised to store the AK and SK in ciphertext in the configuration file or environment variables and decrypt them when needed to ensure security.

{
  "links": [
    {
      "link-config-values": {
        "configs": [
          {
            "inputs": [
              {
                "name": "linkConfig.host",
                "value": "10.120.205.230"
              },
              {
                "name": "linkConfig.authType",
                "value": "KERBEROS"
              },
              {
                "name": "linkConfig.user",
                "value": "cdm"
              },
              {
                "name": "linkConfig.password",
                "value": "Add password here"
              }
            ],
            "name": "linkConfig"
          }
        ]
      },
      "name": "hive_link",
      "connector-name": "hive-connector"
    }
  ]
}

Link Parameters

Parameter	Mandatory	Type	Description
llinkConfig.host	Yes	String	IP address of MRS Manager
linkConfig.authType	Yes	Enumeration	Authentication method of MRS. The options are as follows: SIMPLE: for non-security mode KERBEROS: for security mode
linkConfig.principal	No	String	Account principal required for Kerveros authentication. You can contact the administrator to obtain the account. Before using the cluster configuration, you must set this parameter in cluster configuration management.
linkConfig.keytab	No	FileContent	Local absolute path of the keytab file required for Kerveros authentication. You can also contact the administrator to obtain the file. Before using the cluster configuration, you must set this parameter in cluster configuration management.
linkConfig.hiveVersion	Yes	Enumeration	Hive version HIVE_1_X HIVE_3_X
linkConfig.user	No	String	Username for logging in to Manager. This parameter is not required if the cluster configuration is used.
linkConfig.password	No	String	Password for logging in to Manager. This parameter is not required if the cluster configuration is used.
linkConfig.uri	No	String	NameNode URI required for the link to Apache Hadoop. The format is ip:port.
linkConfig.hiveMsUris	No	String	Hive metadata address for connecting to Apache Hadoop. For details, see the hive.metastore.uris configuration item. Example: thrift://host-192-168-1-212:9083
linkConfig.obsSupport	Yes	Boolean	The server must support OBS storage. When creating a Hive table, you can store the table in OBS.
linkConfig.runMode	Yes	Enumeration	This parameter is used only when the Hive version is HIVE_3_X. Possible values are: EMBEDDED: The link instance runs with CDM. This mode delivers better performance. STANDALONE: The link instance runs in an independent process. If CDM needs to connect to multiple Hadoop data sources (MRS, Hadoop, or CloudTable) with both Kerberos and Simple authentication modes, select STANDALONE or configure different agents. Note: The STANDALONE mode is used to solve the version conflict problem. If the connector versions of the source and destination ends of the same link are different, a JAR file conflict occurs. In this case, you need to place the source or destination end in the STANDALONE process to prevent the migration failure caused by the conflict.
linkConfig.accessKey	No	String	AK. This parameter is mandatory when OBS is supported. You are advised to store it in ciphertext in the configuration file or an environment variable and decrypt it when needed to ensure security.
linkConfig.securityKey	No	String	SK. This parameter is mandatory when OBS is supported. You are advised to store it in ciphertext in the configuration file or an environment variable and decrypt it when needed to ensure security.
linkConfig.properties	No	Map	Properties. You can add configuration properties of the client. Each property must contain a name and a value.

last updated: 2024-11-13 13:47 UTC - commit: db9c60a6c98025320fa32706a5e58cd02cf7d7f1