• MapReduce Service

mrs
  1. Help Center
  2. MapReduce Service
  3. API Reference
  4. APIs
  5. Job Object APIs
  6. Adding a Job and Executing the Job

Adding a Job and Executing the Job

Function

This API is used to add a job to an MRS cluster and execute the job. This API is incompatible with Sahara.

URI

  • Format:

    POST /v1.1/{project_id}/jobs/submit-job

  • Parameter description
    Table 1 URI parameter description

    Parameter

    Mandatory or Not

    Description

    project_id

    Yes

    Project ID. For details on how to obtain the project ID, see Obtaining a Project ID.

Request

  • Example:

    The request example of MapReduce job:

    {
        "job_type": 1, 
        "job_name": "mrs_test_jobone_20170602_141106", 
        "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", 
        "jar_path": "s3a://mrs-opsadm/jarpath/hadoop-mapreduce-examples-2.7.2.jar", 
        "arguments": "wordcount", 
        "input": "s3a://mrs-opsadm/input/", 
        "output": "s3a://mrs-opsadm/output/", 
        "job_log": "s3a://mrs-opsadm/log/", 
        "file_action": "", 
        "hql": "", 
        "hive_script_path": ""
    }

    The request example of Spark job:

    {
        "job_type": 2, 
        "job_name": "mrs_test_sparkjob_20170602_141106", 
        "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", 
        "jar_path": "s3a://mrs-opsadm/jarpath/spark-test.jar", 
        "arguments": "org.apache.spark.examples.SparkPi 10", 
        "input": "", 
        "output": "s3a://mrs-opsadm/output/", 
        "job_log": "s3a://mrs-opsadm/log/", 
        "file_action": "", 
        "hql": "", 
        "hive_script_path": ""
    }

    The request example of Hive Script job:

    {
        "job_type": 3, 
        "job_name": "mrs_test_SparkScriptJob_20170602_141106", 
        "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", 
        "jar_path": "s3a://mrs-opsadm/jarpath/Hivescript.sql", 
        "arguments": "", 
        "input": "s3a://mrs-opsadm/input/", 
        "output": "s3a://mrs-opsadm/output/", 
        "job_log": "s3a://mrs-opsadm/log/", 
        "file_action": "", 
        "hql": "", 
        "hive_script_path": "s3a://mrs-opsadm/jarpath/Hivescript.sql"
    }

    The request example of DistCp job for import:

    {
        "job_type": 5, 
        "job_name": "mrs_test_importjob_20170602_141106", 
        "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", 
        "input": "s3a://mrs-opsadm/jarpath/hadoop-mapreduce-examples-2.7.2.jar", 
        "output": "/user", 
        "file_action": "import"
    }

    The request example of DistCp job for export:

    {
        "job_type": 5, 
        "job_name": "mrs_test_exportjob_20170602_141106", 
        "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", 
        "input": "/user/hadoop-mapreduce-examples-2.7.2.jar", 
        "output": "s3a://mrs-opsadm/jarpath/", 
        "file_action": "export"
    }

    The request example of Spark Script job:

    {
        "job_type": 6, 
        "job_name": "mrs_test_sparkscriptjob_20170602_141106", 
        "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56", 
        "jar_path": "s3a://mrs-opsadm/jarpath/sparkscript.sql", 
        "arguments": "", 
        "input": "s3a://mrs-opsadm/input/", 
        "output": "s3a://mrs-opsadm/output/", 
        "job_log": "s3a://mrs-opsadm/log/", 
        "file_action": "", 
        "hql": "", 
        "hive_script_path": "s3a://mrs-opsadm/jarpath/sparkscript.sql"
    }
  • Parameter description
    Table 2 Request parameter description

    Parameter

    Mandatory or Not

    Type

    Description

    job_type

    Yes

    Integer

    Job type

    • 1: MapReduce
    • 2: Spark
    • 3: Hive Script
    • 4: HiveQL (not supported currently)
    • 5: DistCp, importing and exporting data. For details, see Table 3.
    • 6: Spark Script
    • 7: Spark SQL, submitting Spark SQL statements. For details, see Table 4. (not supported in this API currently)
      NOTE:

      Spark and Hive jobs can be added to only clusters including Spark and Hive components.

    job_name

    Yes

    String

    Job name

    Contains only 1 to 64 letters, digits, hyphens (-), and underscores (_).

    NOTE:

    Identical job names are allowed but not recommended.

    cluster_id

    Yes

    String

    Cluster ID

    jar_path

    Yes

    String

    Path of the .jar package or .sql file for program execution

    The parameter must meet the following requirements:

    • Contains a maximum of 1023 characters, excluding special characters such as ;|&><'$. The address cannot be empty or full of spaces.
    • Starts with / or s3a://.
    • Spark Script must end with .sqlwhile MapReduce and Spark Jar must end with .jarsql and jar are case-insensitive.

    arguments

    No

    String

    Key parameter for program execution. The parameter is specified by the function of the user's program. MRS is only responsible for loading the parameter.

    The parameter contains a maximum of 2047 characters, excluding special characters such as ;|&>'<$, and can be empty.

    NOTE:

    When you enter parameters containing sensitive information, for example, a password for login, you can add an at sign (@) before the parameters to encrypt the parameter values and prevent persistence of sensitive information in the form of plaintext. Therefore, when you view job information on the MRS, sensitive information will be displayed as asterisks (*).

    Example: username=admin @password=admin_123

    input

    No

    String

    Path for inputting data, which must start with / or s3a://. A correct OBS path is required.

    The parameter contains a maximum of 1023 characters, excluding special characters such as ;|&>'<$, and can be empty.

    output

    No

    String

    Path for outputting data, which must start with / or s3a://. A correct OBS path is required. If the path does not exist, the system automatically creates it.

    The parameter contains a maximum of 1023 characters, excluding special characters such as ;|&>'<$, and can be empty.

    job_log

    No

    String

    Path for storing job logs that record job running status. This path must start with / or s3a://. A correct OBS path is required.

    The parameter contains a maximum of 1023 characters, excluding special characters such as ;|&>'<$, and can be empty.

    hive_script_path

    Yes

    String

    SQL program path

    This parameter is needed by Spark Script and Hive Script jobs only and must meet the following requirements:

    • Contains a maximum of 1023 characters, excluding special characters such as ;|&><'$. The address cannot be empty or full of spaces.
    • Starts with / or s3a://.
    • Ends with .sqlsql is case-insensitive.

    is_protected

    No

    Bool

    Whether a job is protected

    • true
    • false

    The current version does not support this function.

    is_public

    No

    Bool

    Whether a job is public

    • true
    • false

    The current version does not support this function.

    Table 3 DistCp parameter description

    Parameter

    Mandatory or Not

    Type

    Description

    job_name

    Yes

    String

    Job name

    Contains only 1 to 64 letters, digits, hyphens (-), and underscores (_).

    NOTE:

    Identical job names are allowed but not recommended.

    input

    No

    String

    Data source path

    • When you import data, the parameter is set to an OBS path.
    • When you export data, the parameter is set to an HDFS path.

    output

    No

    String

    Data receiving path

    • When you import data, the parameter is set to an HDFS path.
    • When you export data, the parameter is set to an OBS path.

    file_action

    Yes

    String

    Types of file operations, including:

    • export: Export data from HDFS to OBS.
    • import: Import data from OBS to HDFS.
    Table 4 Spark SQL parameter description

    Parameter

    Mandatory or Not

    Type

    Description

    hql

    Yes

    String

    Spark SQL statement, which needs Base64 encoding and decoding.

    ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/ is a standard coding table. MRS uses ABCDEFGHILKJMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/ to encode the Spark SQL statement into a Base64 string. The value of the Hql parameter is generated by adding any letter to the beginning of the encoded character string. The Spark SQL statement is generated by decoding the value in the background.

    Example:

    1. On the Web interface, enter the show tables; Spark SQL statement.
    2. Use ABCDEFGHILKJMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/to encode the Spark SQL statement into character string c2hvdyB0YWLsZXM7.
    3. At the beginning of c2hvdyB0YWLsZXM7, add any character, for example gc2hvdyB0YWLsZXM7, to generate the value of the Hql parameter.
    4. The show tables; Spark SQL statement is generated by automatically decoding in the background.

    job_name

    Yes

    String

    Job name, which contains only 1 to 64 letters, digits, hyphens (-), and underscores (_).

    NOTE:

    Identical job names are allowed but not recommended.

    NOTE:

    Files and programs encrypted by the KMS cannot be supported in the OBS path.

Response

  • Example:
    {
      "job_execution": {
        "templated": false,
        "created_at": 1496387588,
        "updated_at": 1496387588,
        "id": "12ee9ae4-6ee1-48c6-bb84-fb0b4f76cf03",
        "tenant_id": "c71ad83a66c5470496c2ed6e982621cc",
        "job_id": "",
        "job_name": "mrs_test_jobone_20170602_141106",
        "input_id": null,
        "output_id": null,
        "start_time": 1496387588,
        "end_time": null,
        "cluster_id": "e955a7a3-d334-4943-a39a-994976900d56",
        "engine_job_id": null,
        "return_code": null,
        "is_public": null,
        "is_protected": null,
        "group_id": "12ee9ae4-6ee1-48c6-bb84-fb0b4f76cf03",
        "jar_path": "s3a://mrs-opsadm/jarpath/hadoop-mapreduce-examples-2.7.2.jar",
        "input": "s3a://mrs-opsadm/input/",
        "output": "s3a://mrs-opsadm/output/",
        "job_log": "s3a://mrs-opsadm/log/",
        "job_type": 1,
        "file_action": "",
        "arguments": "wordcount",
        "hql": "",
        "job_state": 2,
        "job_final_status": 0,
        "hive_script_path": "",
        "create_by": "b67132be2f054a45b247365647e05af0",
        "finished_step": 0,
        "job_main_id": "",
        "job_step_id": "",
        "postpone_at": 1496387588,
        "step_name": "",
        "step_num": 0,
        "task_num": 0,
        "update_by": "b67132be2f054a45b247365647e05af0",
        "credentials": "",
        "user_id": "b67132be2f054a45b247365647e05af0",
        "job_configs": null,
        "extra": null,
        "data_source_urls": null,
        "info": null
      }
    }
  • Parameter description
    Table 5 Response parameter description

    Parameter

    Mandatory or Not

    Type

    Description

    templated

    Yes

    Bool

    Whether job execution objects are generated by job templates

    created_at

    Yes

    Integer

    Creation time, which is a 10-bit timestamp.

    updated_at

    Yes

    Integer

    Update time, which is a 10-bit timestamp.

    id

    Yes

    String

    Job ID

    tenant_id

    Yes

    String

    Project ID. For details on how to obtain the project ID, see Obtaining a Project ID.

    job_id

    Yes

    String

    Job application ID

    job_name

    Yes

    String

    Job name

    input_id

    Yes

    String

    Null

    output_id

    Yes

    String

    Null

    start_time

    Yes

    Integer

    Start time of job execution, which is a 10-bit timestamp.

    end_time

    Yes

    Integer

    End time of job execution, which is a 10-bit timestamp.

    cluster_id

    Yes

    String

    Cluster ID

    engine_job_id

    Yes

    String

    Workflow ID of Oozie

    return_code

    Yes

    Integer

    Returned code for an execution result

    is_public

    Yes

    Bool

    Whether a job is public

    • true
    • false

    The current version does not support this function.

    is_protected

    Yes

    Bool

    Whether a job is protected

    • true
    • false

    The current version does not support this function.

    group_id

    Yes

    String

    Group ID of a job

    jar_path

    Yes

    String

    Path of the .jar file for program execution

    input

    Yes

    String

    Address for inputting data

    output

    Yes

    String

    Address for outputting data

    job_log

    Yes

    String

    Address for storing job logs

    job_type

    Yes

    Integer

    Job type

    • 1: MapReduce
    • 2: Spark
    • 3: Hive Script
    • 4: HiveQL (not supported temporarily)
    • 5: DistCp
    • 6: Spark Script
    • 7: Spark SQL (not supported in this API currently)

    file_action

    Yes

    String

    Data import and export

    arguments

    Yes

    String

    Key parameter for program execution. The parameter is specified by the function of the user's internal program. MRS is only responsible for loading the parameter. This parameter can be empty.

    job_state

    Yes

    Integer

    Job status

    • -1: Terminated
    • 1: Starting
    • 2: Running
    • 3: Completed
    • 4: Abnormal
    • 5: Error

    job_final_status

    Yes

    Integer

    Final job status

    • 0: unfinished
    • 1: terminated due to an execution error
    • 2: executed successfully
    • 3: canceled

    hive_script_path

    Yes

    String

    Hive script address

    create_by

    Yes

    String

    User ID for creating jobs

    This parameter is not used in the current version, but is retained for compatibility with earlier versions.

    finished_step

    Yes

    Integer

    Number of completed steps

    This parameter is not used in the current version, but is retained for compatibility with earlier versions.

    job_main_id

    Yes

    String

    Main ID of a job

    This parameter is not used in the current version, but is retained for compatibility with earlier versions.

    job_step_id

    Yes

    String

    Step ID of a job

    This parameter is not used in the current version, but is retained for compatibility with earlier versions.

    postpone_at

    Yes

    Integer

    Delay time, which is a 10-bit timestamp.

    This parameter is not used in the current version, but is retained for compatibility with earlier versions.

    step_name

    Yes

    String

    Step name of a job

    This parameter is not used in the current version, but is retained for compatibility with earlier versions.

    step_num

    Yes

    Integer

    Number of steps

    This parameter is not used in the current version, but is retained for compatibility with earlier versions.

    task_num

    Yes

    Integer

    Number of tasks

    This parameter is not used in the current version, but is retained for compatibility with earlier versions.

    update_by

    Yes

    String

    User ID for updating jobs

    credentials

    Yes

    String

    Token

    The current version does not support this function.

    user_id

    Yes

    String

    User ID for creating jobs

    This parameter is not used in the current version, but is retained for compatibility with earlier versions.

    job_configs

    Yes

    String

    Key-value pair set for saving job running configurations

    extra

    Yes

    String

    Authentication information

    The current version does not support this function.

    data_source_urls

    Yes

    String

    Data source URL of a job

    info

    Yes

    String

    Key-value pair set, containing job running information returned by Oozie

Status Code

Table 6 describes the status code of this API.

Table 6 Status Code

Status Code

Description

200

The job is successfully added.

For the description about error status codes, see section Status Codes.