Common Loader Parameters¶
Parameter Description¶
Parameter | Description | Default Value | Value Range |
|---|---|---|---|
mapreduce.client.submit.file.replication | Number of copies of the job files that the MapReduce task depends on in HDFS. If the number of DataNodes in the cluster is less than the value of this parameter, the number of copies is equal to the number of DataNodes. If the number of DataNodes is greater than or equal to the value of this parameter, the number of copies is the value of this parameter. | 10 | 3 to 256 |
loader.fault.tolerance.rate | Error tolerance. If the value is greater than 0, the error tolerance mechanism is enabled. When enabling the fault tolerance mechanism, you are advised to set the number of Map jobs to be greater than or equal to 3. It is recommended that this function be used when the job data volume is large. | 0 | 0 to 1.0 |
loader.input.field.separator | Default input field separator. The parameter value takes effect only when input and output conversion steps are configured. The conversion steps can be left blank. If no separators are configured in job conversion steps, the default separator is used. | , |
|
loader.input.line.separator | Default input line separator. The parameter value takes effect only when input and output conversion steps are configured. The conversion steps can be left blank. If no separators are configured in job conversion steps, the default separator is used. |
|
|
loader.output.field.separator | Default output field separator. The parameter value takes effect only when input and output conversion steps are configured. The conversion steps can be left blank. If no separators are configured in job conversion steps, the default separator is used. | , |
|
loader.output.line.separator | Line separator of data that Loader outputs |
|
|
Note
Because it needs time to calculate the fault tolerance rate, you are recommended to use the loader.fault.tolerance.rate parameter when the job runtime is longer than 2 minutes to ensure user experience.
Default separators are configured for the parameters in the preceding table for Loader. If separators are configured in the conversion steps for the jobs, the separators in the conversion steps will be used. If separators are not configured in the conversion steps, the default separators will be used.