• Data Ingestion Service

dis
  1. Help Center
  2. Data Ingestion Service
  3. API Reference
  4. API Description
  5. Creating a DIS Stream

Creating a DIS Stream

Function

This API is used to create a DIS stream.

URL

  • URL format

    POST /v2/{project_id}/streams

  • Parameter description

    None

Request

  • Example request
    • Create a stream whose source data type is BLOB. Dump tasks cannot be created for such a stream.
      POST https://{endpoint}:{port}/v2/6352bba7aaab443aa1d9943efc586a68/streams      
      Request Header:  
      Content-Type: application/json   
      X-Sdk-Date: 20180820T024248Z    
      Authorization: SDK-HMAC-SHA256   
      Credential=QRUP2R3QFNAOVAWMYHZW/20180820/
      eu-de/test/sdk_request,
      SignedHeaders=host;x-sdk-date, 
      Signature=9d8b56b055c0e1f7a9498d881a7cb726be91b4f0cde1773b0b1557e987a480ce
      {  
      	"stream_name": "dis-DLpR",
      	"partition_count": 1,
      	"stream_type": "COMMON",
      	"data_type": "BLOB",
      	"data_duration": 24
      } 
    • Create a stream whose source data type is JSON or CSV. Dump tasks cannot be created for such a stream.
      POST https://{endpoint}:{port}/v2/6352bba7aaab443aa1d9943efc586a68/streams       
      Request Header:  
      Content-Type: application/json   
      X-Sdk-Date: 20180820T024248Z    
      Authorization: SDK-HMAC-SHA256   
      Credential=QRUP2R3QFNAOVAWMYHZW/20180820/
      eu-de/test/sdk_request,
      SignedHeaders=host;x-sdk-date, 
      Signature=9d8b56b055c0e1f7a9498d881a7cb726be91b4f0cde1773b0b1557e987a480ce
      {  
      	"stream_name": "dis-DLpR",
      	"partition_count": 1,
      	"stream_type": "COMMON",
              "data_type": "JSON",
      	"data_duration": 24,
              "data_schema":"{\"type\":\"record\",\"name\":\"RecordName\",\"fields\":[{\"name\":\"id\",\"type\":\"string\",\"doc\":\"Type inferred from '\\\"2017/10/11 11:11:11\\\"'\"},{\"name\":\"info\",\"type\":{\"type\":\"array\",\"items\":{\"type\":\"record\",\"name\":\"info\",\"fields\":[{\"name\":\"date\",\"type\":\"string\",\"doc\":\"Type inferred from '\\\"2018/10/11 11:11:11\\\"'\"}]}},\"doc\":\"Type inferred from '[{\\\"date\\\":\\\"2018/10/11 11:11:11\\\"}]'\"}]}"}
      }

  • Parameter description
Table 1 Parameter description

Parameter

Mandatory

Type

Description

stream_name

Yes

String

Name of the DIS stream.

Each DIS stream has a unique name. A stream name is 1 to 64 characters in length. Only letters, digits, hyphens (-), and underscores (_) are allowed.

stream_type

No

String

Stream type. Possible values:

  • Common: indicates a common stream. The bandwidth is 1 MB/s.
  • ADVANCED: indicates an advanced stream. The bandwidth is 5 MB/s.

Default value: COMMON

partition_count

Yes

Int

Quantity of the partitions into which data records in the newly created DIS stream will be distributed.

Partitions are the base throughput unit of a DIS stream.

The value range varies depending on the value of stream_type.

  • If stream_type is not specified or set to COMMON, the value of partition_count is an integer from 1 to 50. If the tenant has created N common partitions, the maximum value of partition_count is 50-N.
  • If stream_type is set to ADVANCED, the value of partition_count is an integer from 1 to 10. If the tenant has created N advanced partitions, the maximum value of partition_count is 10-N.

data_duration

No

Int

Period of time for which data is retained in the DIS stream.

Value range: N x 24, where N is an integer from 1 to 7.

Unit: hour

Default value: 24

If this parameter is left unspecified, the default value will be used.

data_schema

No

String

Source data structure that defines JOSN and CSV formats. It is described in the syntax of Avro. For details about Avro, see http://avro.apache.org/docs/current/#schemas.

NOTE:

This parameter is mandatory when Dump File Format is parquet or carbon.

tags

No

List<Tag>

Label of the stream.

obs_destination_descriptor

No

Object

Parameter list of the OBS to which data in the DIS stream will be dumped. Data in a DIS stream cannot be dumped to multiple destinations.

By default, this parameter is left unspecified, indicating that data in the DIS stream will not be dumped to OBS.

NOTE:

This parameter is available and needs to be configured only when Source Data Type is set to FILE.

Table 2 obs_destination_descriptor parameter description

Parameter

Mandatory

Type

Description

agency_name

Yes

String

Name of the agency created in IAM. DIS uses an agency to access your specified resources.

IAM agency parameter settings:

  • Agency Type: Cloud service
  • Cloud Service: DIS
  • Validity Period: Permanent
  • Set Policy to Tenant Administrator on the OBS project in the Global service region.

This parameter cannot be left unspecified and the parameter value cannot exceed 64 characters.

obs_bucket_path

Yes

String

Name of the OBS bucket used to store data from the DIS stream.

Table 3 Tag parameter description

Parameter

Mandatory

Type

Description

key

Yes

String

Key. A tag key cannot contain special characters such as =*<>\,|/ or start or end with a space.

value

Yes

String

Value. A tag value cannot contain special characters such as =*<>\,|/ or start or end with a space.

Response

  • If the DIS stream was successfully created, a 201 response with an empty response body is returned.
  • If the DIS stream failed to be created, identify the failure cause according to the response body and the instructions in Error Codes.

Status Code

  • Normal

    201 Created

  • Failed

    For more information, see Error Codes.