Adding OBS Dump Tasks¶

Function¶

This API is used to add OBS dump tasks.

Calling Method¶

For details, see Calling APIs.

URI¶

POST /v2/{project_id}/streams/{stream_name}/transfer-tasks

**Table 1** Path Parameters¶
Parameter	Mandatory	Type	Description
project_id	Yes	String	Project ID
stream_name	Yes	String	Name of the stream Maximum: 60

Request Parameters¶

**Table 2** Request header parameters¶
Parameter	Mandatory	Type	Description
X-Auth-Token	Yes	String	User token. It can be obtained by calling the IAM API used to obtain a user token. The value of X-Subject-Token in the response header is the user token.

**Table 3** Request body parameters¶
Parameter	Mandatory	Type	Description
destination_type	Yes	String	Type of the dump task. OBS: Data is dumped to OBS. MRS: Data is dumped to MRS. DLI: Data is dumped to DLI. CLOUDTABLE: Data is dumped to CloudTable. DWS: Data is dumped to DWS. Default: NOWHERE Enumeration values: OBS
obs_destination_descriptor	No	OBSDestinationDescriptorRequest object	Parameter list of OBS to which data in the DIS stream will be dumped

**Table 4** OBSDestinationDescriptorRequest¶
Parameter	Mandatory	Type	Description
task_name	Yes	String	Name of the dump task. The task name consists of letters, digits, hyphens (-), and underscores (_). It contains 1 to 64 characters.
agency_name	Yes	String	Name of the agency created on IAM. DIS uses an agency to access your specified resources. Agency parameter settings:- Agency Type: Cloud service- Cloud Service: DIS- Validity Period: Unlimited- Set Policy to Tenant Administrator on the OBS project in the Global service region.If agencies are available, you can use an IAM API to obtain the available agencies.This parameter cannot be left unspecified and the parameter value cannot exceed 64 characters.If there are dump tasks on the console, the system displays a message indicating that an agency will be automatically created. The name of the automatically created agency is dis_admin_agency. Maximum: 64
deliver_time_interval	Yes	Integer	User-defined interval at which data is imported from the current DIS stream into OBS. If no data is pushed to the DIS stream during the current interval, no dump file package will be generated. Unit: second Minimum: 30 Maximum: 900 Default: 300
consumer_strategy	No	String	Offset. LATEST: maximum offset, indicating that the latest data will be extracted. TRIM_HORIZON: minimum offset, indicating that the earliest data will be extracted. Default: LATEST Enumeration values: LATEST TRIM_HORIZON
file_prefix	No	String	Directory to store files that will be dumped to OBS. Different directory levels are separated by slashes (/) and cannot start with slashes. The value can contain a maximum of 50 characters, including letters, digits, underscores (_), and slashes (/). This parameter is left blank by default. Maximum: 50
partition_format	No	String	Directory structure of the object file written into OBS. The directory structure is in the format of yyyy/MM/dd/HH/mm (time at which the dump task was created).- N/A: If this parameter is left blank, the time directory format will not be used.- yyyy: year.- yyyy/MM: year and month.- yyyy/MM/dd: year, month, and day.- yyyy/MM/dd/HH: year, month, day, and hour.- yyyy/MM/dd/HH/mm: year, month, day, hour, and minute.For example, if the dump task was created at 14:49 on November 10, 2017, then the directory structure is 2017 > 11 > 10 > 14 > 49.Default value: emptyNote:After the data is dumped successfully, the storage directory structure is obs_bucket_path/file_prefix/partition_format. Enumeration values: yyyy yyyy/MM yyyy/MM/dd yyyy/MM/dd/HH yyyy/MM/dd/HH/mm
obs_bucket_path	Yes	String	Name of the OBS bucket used to store the stream data
destination_file_type	No	String	Dump file format. text: This is the default value. parquet carbon Note: The parquet or carbon format can be selected only when Source Data Type is set to JSON and Dump Destination is set to OBS. Default: text Enumeration values: text parquet carbon
processing_schema	No	ProcessingSchema object	Dump time directory generated based on the timestamp of the source data and the configured partition_format. Directory structure of the object file written into OBS. The directory structure is in the format of yyyy/MM/dd/HH/mm.
record_delimiter	No	String	Delimiter for the dump file, which is used to separate the user data that is written into the dump file. Options: Comma (,): default value Semicolon (;) Vertical bar (\|) Newline (\n) Default: \n

**Table 5** ProcessingSchema¶
Parameter	Mandatory	Type	Description
timestamp_name	Yes	String	Attribute name of the source data timestamp
timestamp_type	Yes	String	Type of the source data timestamp. String Timestamp: 13-bit timestamp of the long type
timestamp_format	No	String	OBS directory generated based on the timestamp format. This parameter is mandatory when the timestamp type of the source data is String. Enumeration values: yyyy/MM/dd HH:mm:ss MM/dd/yyyy HH:mm:ss dd/MM/yyyy HH:mm:ss yyyy-MM-dd HH:mm:ss MM-dd-yyyy HH:mm:ss dd-MM-yyyy HH:mm:ss

Response Parameters¶

None

Example Requests¶

Adding OBS Dump Tasks

POST https://{Endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks

{
  "destination_type" : "OBS",
  "obs_destination_descriptor" : {
    "task_name" : "newtask",
    "consumer_strategy" : "LATEST",
    "agency_name" : "dis_admin_agency",
    "destination_file_type" : "text",
    "obs_bucket_path" : "obsbucket",
    "file_prefix" : "",
    "partition_format" : "yyyy/MM/dd/HH/mm",
    "record_delimiter" : "|",
    "deliver_time_interval" : 30
  }
}

Adding OBS Dump Tasks (The dump file format is Parquet.)

POST https://{Endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks

{
  "destination_type" : "OBS",
  "obs_destination_descriptor" : {
    "task_name" : "newtask",
    "consumer_strategy" : "LATEST",
    "agency_name" : "dis_admin_agency",
    "destination_file_type" : "parquet",
    "obs_bucket_path" : "obsbucket",
    "file_prefix" : "",
    "partition_format" : "yyyy/MM/dd/HH/mm",
    "record_delimiter" : "|",
    "deliver_time_interval" : 30
  }
}

Example Responses¶

None

Status Codes¶

Status Code	Description
201	Normal response

Error Codes¶

See Error Codes.

last updated: 2025-01-27 10:23 UTC - commit: 90a70290da03409ec489fa6c4ae2059aaf4365fe