Creating and Submitting a Flink Job¶
Scenario Description¶
This section describes how to create and run a user-defined Flink job using APIs.
Constraints¶
It takes 6 to 10 minutes to start a job using a new queue for the first time.
Involved APIs¶
Creating a Queue: Create a queue.
Uploading a Package Group (Discarded): Upload the resource package required by the Flink custom job.
Querying Resource Packages in a Group (Discarded): Check whether the uploaded resource package is correct.
Creating a Flink Jar job Create a user-defined Flink job.
Running Jobs in Batches: Run a user-defined Flink job.
Procedure¶
Create a queue for general use. For details, see Creating a Queue. In the request, set resource_mode to 1 to create a dedicated queue.
Upload the resource package of the user-defined Flink job. For details, see 2.
Query resource packages in a group. For details, see 3.
Create a custom flink job.
API
URI format: POST /v1.0/{project_id}/streaming/flink-jobs
Obtain the value of {project_id} from Obtaining a Project ID.
For details about the request parameters, see Creating a Database (Discarded).
Request example
Description: Create a user-defined Flink job in the project whose ID is 48cc2c48765f481480c7db940d6409d1.
Example URL: POST https://{endpoint}/v1.0/48cc2c48765f481480c7db940d6409d1/streaming/flink-jobs
Body:
{ "name": "test", "desc": "job for test", "queue_name": "testQueue", "manager_cu_number": 1, "cu_number": 2, "parallel_number": 1, "tm_cus": 1, "tm_slot_num": 1, "log_enabled": true, "obs_bucket": "bucketName", "smn_topic": "topic", "main_class": "org.apache.flink.examples.streaming.JavaQueueStream", "restart_when_exception": false, "entrypoint": "javaQueueStream.jar", "entrypoint_args":"-windowSize 2000 -rate3", "dependency_jars": [ "myGroup/test.jar", "myGroup/test1.jar" ], "dependency_files": [ "myGroup/test.csv", "myGroup/test1.csv" ] }
Example response
{ "is_success": true, "message": "A Flink job is created successfully.", "job": { "job_id": 138, "status_name": "job_init", "status_desc": "" } }
Run jobs in batches.
API
URI format: POST /v1.0/{project_id}/streaming/jobs/run
Obtain the value of {project_id} from Obtaining a Project ID.
For details about the request parameters, see Running Jobs in Batches.
Request example
Description: Run the jobs whose job_id is 298765 and 298766 in the project whose ID is 48cc2c48765f481480c7db940d6409d1.
Example URL: POST https://{endpoint}/v1.0/48cc2c48765f481480c7db940d6409d1/streaming/jobs/run
Body:
{ "job_ids": [131,130,138,137], "resume_savepoint": true }
Example response
[ { "is_success": "true", "message": "The request for submitting DLI jobs is delivered successfully." }, { "is_success": "true", "message": "The request for submitting DLI jobs is delivered successfully." }, { "is_success": "true", "message": "The request for submitting DLI jobs is delivered successfully." }, { "is_success": "true", "message": "The request for submitting DLI jobs is delivered successfully." } ]