Data Ingestion Service (DIS)¶

The DIS client is the command-line interface (CLI) for the Data Ingestion Service (DIS) API and its extensions.

For help on a specific dis command, enter:

$ openstack dis help SUBCOMMAND

DIS App Operations¶

dis app consumption list¶

List App Consumptions.

openstack dis app consumption list
    [--sort-column SORT_COLUMN]
    [--sort-ascending | --sort-descending]
    [--limit <limit>]
    [--start-partition-id <start_partition_id>]
    [--checkpoint-type <checkpoint_type>]
    <streaName>
    <appName>

--sort-column SORT_COLUMN¶: specify the column(s) to sort the data (columns specified first have a priority, non-existing columns are ignored), can be repeated

--sort-ascending¶: sort the column(s) in ascending order

--sort-descending¶: sort the column(s) in descending order

--limit <limit>¶: Maximum number of apps to list in a single API call. Default: 10

--start-partition-id <start_partition_id>¶: Name of the partition to start the partition list with. The returned partition list does not contain this partition.

--checkpoint-type <checkpoint_type>¶: Type of the checkpoint. LAST_READ: Only sequence numbers are recorded in databases.

streaName¶: Name of the stream to be queried.

appName¶: Name of the app to be queried.

This command is provided by the otcextensions plugin.

dis app create¶

Create new Consumption App.

openstack dis app create <appName>

appName¶: Specifies the name of the App.

This command is provided by the otcextensions plugin.

dis app delete¶

Delete Consumption App(s).

openstack dis app delete <appName> [<appName> ...]

appName¶: Name of Dis App(s) to delete.

This command is provided by the otcextensions plugin.

dis app list¶

Query List of Apps.

openstack dis app list
    [--sort-column SORT_COLUMN]
    [--sort-ascending | --sort-descending]
    [--limit <limit>]
    [--start-app-name <start_app_name>]
    [--stream-name <stream_name>]

--sort-column SORT_COLUMN¶: specify the column(s) to sort the data (columns specified first have a priority, non-existing columns are ignored), can be repeated

--sort-ascending¶: sort the column(s) in ascending order

--sort-descending¶: sort the column(s) in descending order

--limit <limit>¶: Maximum number of apps to list in a single API call. Default: 10

--start-app-name <start_app_name>¶: Name of the app to start the list with. The returned app list does not contain this app name.

--stream-name <stream_name>¶: Name of the stream whose apps will be returned.

This command is provided by the otcextensions plugin.

dis app show¶

Querying App Details.

openstack dis app show <appName>

appName¶: Name of the app to be queried.

This command is provided by the otcextensions plugin.

DIS Checkpoint Operations¶

dis checkpoint create¶

Add Checkpoint.

openstack dis checkpoint create
    --partition-id <partition_id>
    --sequence-number <sequence_number>
    [--metadata <metadata>]
    [--checkpoint-type {LAST_READ}]
    <streamName>
    <appName>

--partition-id <partition_id>¶: Identifier of the stream partition to which the checkpoint belongs.

--sequence-number <sequence_number>¶: Sequence number to be submitted, which is used to record the consumption checkpoint of the stream.

--metadata <metadata>¶: Metadata information of the consumer application. The metadata information can contain a maximum of 1,000 characters.

--checkpoint-type {LAST_READ}¶: Type of the checkpoint. LAST_READ: Only sequence numbers are recorded indatabases. Default: LAST_READ

streamName¶: Name of the stream to which the checkpoint belongs.

appName¶: Name of the app associated with the checkpoint.

This command is provided by the otcextensions plugin.

dis checkpoint delete¶

Delete Checkpoint.

openstack dis checkpoint delete
    [--partition-id <partition_id>]
    [--checkpoint-type {LAST_READ}]
    <streamName>
    <appName>

--partition-id <partition_id>¶: Identifier of the stream partition to which the checkpoint belongs.

--checkpoint-type {LAST_READ}¶: Type of the checkpoint. LAST_READ: Only sequence numbers are recorded indatabases. Default: LAST_READ

streamName¶: Name of the stream to which the checkpoint belongs.

appName¶: Name of the app associated with the checkpoint.

This command is provided by the otcextensions plugin.

dis checkpoint show¶

Querying Checkpoint Details.

openstack dis checkpoint show
    --partition-id <partition_id>
    [--checkpoint-type {LAST_READ}]
    <streamName>
    <appName>

--partition-id <partition_id>¶: Identifier of the stream partition to which the checkpoint belongs.

--checkpoint-type {LAST_READ}¶: Type of the checkpoint. LAST_READ: Only sequence numbers are recorded indatabases. Default: LAST_READ

streamName¶: Name of the stream to which the checkpoint belongs.

appName¶: Name of the app associated with the checkpoint.

This command is provided by the otcextensions plugin.

DIS Data Operations¶

dis data cursor show¶

Query Data Cursor.

openstack dis data cursor show
    --partition-id <partition_id>
    [--cursor-type {AT_SEQUENCE_NUMBER,AFTER_SEQUENCE_NUMBER,TRIM_HORIZON,LATEST,AT_TIMESTAMP}]
    [--starting-seq-num <starting_seq_num>]
    [--timestamp <timestamp>]
    [--stream-id <stream_id>]
    <streamName>

--partition-id <partition_id>¶: Partition ID of the stream.

--cursor-type {AT_SEQUENCE_NUMBER,AFTER_SEQUENCE_NUMBER,TRIM_HORIZON,LATEST,AT_TIMESTAMP}¶: Cursor Type.

--starting-seq-num <starting_seq_num>¶: Sequence number. A sequence number is the unique identifier of each record.

--timestamp <timestamp>¶: Timestamp when the data record starts to be read, which is closely related to cursor type AT_TIMESTAMP.

--stream-id <stream_id>¶: Unique ID of the stream.

streamName¶: Name of the stream.

This command is provided by the otcextensions plugin.

dis data download¶

Download Data.

openstack dis data download
    [--sort-column SORT_COLUMN]
    [--sort-ascending | --sort-descending]
    --partition-cursor <partition_cursor>
    [--max-fetch-bytes <max_fetch_bytes>]
    [--filename <filename>]

--sort-column SORT_COLUMN¶: specify the column(s) to sort the data (columns specified first have a priority, non-existing columns are ignored), can be repeated

--sort-ascending¶: sort the column(s) in ascending order

--sort-descending¶: sort the column(s) in descending order

--partition-cursor <partition_cursor>¶: Data cursor, which needs to be obtained through the API for obtaining data cursors. Value: 1 to 512 characters. Note: The validity period of a data cursor is 5 minutes. Maximum number of apps to list in a single API call.

--max-fetch-bytes <max_fetch_bytes>¶: Maximum number of bytes that can be obtained for each request. Note: If the value is less than the size of a single record in the partition, the record cannot be obtained.

--filename <filename>¶: To save data to a file. Note: File exentension should be csv

This command is provided by the otcextensions plugin.

dis data file template¶

Print Data File Template.

openstack dis data file template --filename <filename>

--filename <filename>¶: FileName to generate the template.

This command is provided by the otcextensions plugin.

dis data upload¶

upload data to DIS streams.

openstack dis data upload
    [--stream-id <stream_id>]
    [--data <data>]
    [--explicit-hash-key <explicit_hash_key>]
    [--partition-id <partition_id>]
    [--partition-key <partition_key>]
    [--filename <filename>]
    <streamName>

--stream-id <stream_id>¶: Unique ID of the stream. If no stream is found based on stream_name and stream_id is not empty, stream_id is used to search for the stream.

--data <data>¶: Data to be uploaded. The uploaded data is the serialized binary data (character string encoded using Base64).

--explicit-hash-key <explicit_hash_key>¶: Hash value of the data to be written to the partition. The hash value overwrites the hash value of partition_key.

--partition-id <partition_id>¶: Partition ID of the stream.

--partition-key <partition_key>¶: Partition to which data is written to. Note:If the partition_id parameter is transferred, it will be preferentially used. If partition_id is not passed, partition_key will be used.

--filename <filename>¶: Data file path in CSV format. To get template of a data file run this command: openstack dis data file template

streamName¶: Name of the Stream.

This command is provided by the otcextensions plugin.

DIS Dump Task Operations¶

dis dump task create¶

Adding a Dump Task.

openstack dis dump task create
    [--destination-type <destrination_type>]
    --agency-name <agency_name>
    --deliver-time-interval <deliver_time_interval>
    [--consumer-strategy {LATEST,TRIM_HORIZON}]
    [--file-prefix <file_prefix>]
    [--partition-format {yyyy,yyyy/MM,yyyy/MM/dd,yyyy/MM/dd/HH,yyyy/MM/dd/HH/mm}]
    --obs-bucket-path <obs_bucket_path>
    [--destination-file-type <destination_file_type>]
    [--record-delimiter {,,;,|,\n}]
    <streamName>
    <taskName>

--destination-type <destrination_type>¶: Specifies the name of the DIS Stream.

--agency-name <agency_name>¶: Name of the agency created on IAM. DIS uses an agency to access your specified resources.

--deliver-time-interval <deliver_time_interval>¶: User-defined interval at which data is imported from the current DIS stream into OBS.

--consumer-strategy {LATEST,TRIM_HORIZON}¶: Offset. LATEST: Maximum offset, indicating that the latest data will be extracted. TRIM_HORIZON: Minimum offset, indicating that the earliest data will be extracted. Default value: LATEST.

--file-prefix <file_prefix>¶: Directory to store files that will be dumped to OBS. Different directory levels are separated by slashes (/) and cannot start with slashes.

--partition-format {yyyy,yyyy/MM,yyyy/MM/dd,yyyy/MM/dd/HH,yyyy/MM/dd/HH/mm}¶: Directory structure of the object file written into OBS. The directory structure is in the format of yyyy/MM/dd/HH/mm (time at which the dump task was created). N/A: Leave this parameter empty, indicating that the date and time directory is not used.

--obs-bucket-path <obs_bucket_path>¶: Name of the OBS bucket used to store data from the DIS stream.

--destination-file-type <destination_file_type>¶: Dump file format. Possible values:Text (default)

--record-delimiter {,,;,|,\n}¶: Delimiter for the dump file, which is used to separate the user data that is written into the dump file. Default: n

streamName¶: Specifies the name of the DIS Stream.

taskName¶: Name of the dump task.

This command is provided by the otcextensions plugin.

dis dump task delete¶

Deletes DIS Dump Task(s).

openstack dis dump task delete <streamName> <taskName> [<taskName> ...]

streamName¶: Name of Dis Stream.

taskName¶: Name of Dump Task(s) to delete.

This command is provided by the otcextensions plugin.

dis dump task list¶

List Dump Tasks.

openstack dis dump task list
    [--sort-column SORT_COLUMN]
    [--sort-ascending | --sort-descending]
    <streamName>

--sort-column SORT_COLUMN¶: specify the column(s) to sort the data (columns specified first have a priority, non-existing columns are ignored), can be repeated

--sort-ascending¶: sort the column(s) in ascending order

--sort-descending¶: sort the column(s) in descending order

streamName¶: Specifies the name of the DIS Stream.

This command is provided by the otcextensions plugin.

dis dump task pause¶

Pause Dump Task(s).

openstack dis dump task pause <streamName> <taskId> [<taskId> ...]

streamName¶: Name of Dis Stream.

taskId¶: ID of Dump Task(s) to Pause.

This command is provided by the otcextensions plugin.

dis dump task show¶

Query Details of a Dump Task.

openstack dis dump task show <streamName> <taskName>

streamName¶: Specifies the name of the DIS Stream.

taskName¶: Specifies the name of Dump Task Name.

This command is provided by the otcextensions plugin.

dis dump task start¶

Start Dump Task(s).

openstack dis dump task start <streamName> <taskId> [<taskId> ...]

streamName¶: Name of Dis Stream.

taskId¶: ID of Dump Task(s) to Start.

This command is provided by the otcextensions plugin.

DIS Stream Operations¶

dis stream create¶

Create new DIS Stream.

openstack dis stream create
    --partition-count <partition_count>
    [--stream-type {COMMON,ADVANCED}]
    [--data-type {BLOB}]
    [--data-duration <data_duration>]
    [--autoscale]
    [--autoscale-min-count <autoscale_min_count>]
    [--autoscale-max-count <autoscale_max_count>]
    [--compression-format {snappy,gzip,zip}]
    [--tag key=<key>,value=<value>]
    <streamName>

--partition-count <partition_count>¶: Number of partitions. Partitions are the base throughput unit of the DIS stream.

--stream-type {COMMON,ADVANCED}¶: Stream type. Supported Types: COMMON: a common stream with a bandwidth of 1 MB/s. ADVANCED: an advanced stream with a bandwidth of 5 MB/s. Default value: COMMON.

--data-type {BLOB}¶: Source data type. Supported Types: BLOB: a collection of binary data stored as a single entity in a database management system. Default value: BLOB.

--data-duration <data_duration>¶: Data retention period. Value range: 24–72 Unit: hour Default value: 24

--autoscale¶: Whether to enable auto scaling.

--autoscale-min-count <autoscale_min_count>¶: Minimum number of partitions for automatic scale-down when auto scaling is enabled. Default: 1

--autoscale-max-count <autoscale_max_count>¶: Maximum number of partitions for automatic scale-up when auto scaling is enabled. Default: 1

--compression-format {snappy,gzip,zip}¶: Data compression type. The following types are available: snappy gzip zip Data is not compressed by default.

--tag key=<key>,value=<value>¶: Add Tag(s) to a Stream. key=<key>: Tag key. The value can contain 1 to 36 characters. Only digits, letters, hyphens (-) and underscores (_) are allowed. value=<value>: Tag value. The value can contain 0 to 43 characters. Only digits, letters, hyphens (-) and underscores (_) are allowed.

streamName¶: Specify the name of the DIS Stream.

This command is provided by the otcextensions plugin.

dis stream delete¶

Deletes DIS Stream.

openstack dis stream delete <streamName> [<streamName> ...]

streamName¶: Name of Dis Stream(s) to delete.

This command is provided by the otcextensions plugin.

dis stream list¶

List Dis Streams.

openstack dis stream list
    [--sort-column SORT_COLUMN]
    [--sort-ascending | --sort-descending]
    [--limit <limit>]
    [--start-stream-name <start_stream_name>]

--sort-column SORT_COLUMN¶: specify the column(s) to sort the data (columns specified first have a priority, non-existing columns are ignored), can be repeated

--sort-ascending¶: sort the column(s) in ascending order

--sort-descending¶: sort the column(s) in descending order

--limit <limit>¶: Limit to fetch number of records.

--start-stream-name <start_stream_name>¶: Limit to fetch number of records.

This command is provided by the otcextensions plugin.

dis stream show¶

Show DIS Stream details.

openstack dis stream show <streamName>

streamName¶: Specifies the name of the DIS Stream.

This command is provided by the otcextensions plugin.

dis stream update partition¶

Update Partition Quantity of a DIS Stream.

openstack dis stream update partition
    --partition-count <partition_count>
    <streamName>

--partition-count <partition_count>¶: Number of the target partitions.The value is an integer greater than 0. Each stream can be scaled up and down for five times within one hour. After a stream is scaled up or down, it cannot be scaled up or down again in the next one hour.

streamName¶: Specifies the Name of the DIS Stream.

This command is provided by the otcextensions plugin.

last updated: 2024-05-18 00:15