DIS API¶

The Data Ingestion Service Class¶

The dis high-level interface is available through the dis member of a Connection object. The dis member will only be added if the otcextensions.sdk.register_otc_extensions(conn) method is called.

App Operations¶

class otcextensions.sdk.dis.v2._proxy.Proxy(session, *args, **kwargs)

create_app(**attrs)

Create an App from attributes

Parameters:: attrs (dict) – Keyword arguments which will be used to create a App, comprised of the properties on the App class.
Returns:: The results of app creation
Return type:: App

apps(**params)

List all Apps.

Parameters:: params (dict) – Optional query parameters to be sent to limit the apps being returned.
Returns:: a generator of (App) instances

get_app(app_name)

Query details of a DIS App.

Parameters:: app_name – Name of the app to be queried.
Returns:: One App
Raises:: ResourceNotFound when no resource can be found.

app_consumptions(stream_name, app_name, **params)

List partition consuming state list.

Parameters:

stream_name – Name of the stream.
app_name – Name of the app to be queried.
params (dict) – Optional query parameters to be sent to limit the app consumptions being returned.

Returns:

a generator of (App) instances

delete_app(app_name, ignore_missing=False)

Delete a DIS App.

Parameters:

app_name – The value can be the name of a DIS app.
ignore_missing (bool) – When set to False ResourceNotFound will be raised when the app does not exist. When set to True, no exception will be set when attempting to delete a nonexistent app.

Returns:

None

Checkpoint Operations¶

class otcextensions.sdk.dis.v2._proxy.Proxy(session, *args, **kwargs)

create_checkpoint(**attrs)

Add a Checkpoint from attributes.

Parameters:: attrs (dict) – Keyword arguments which will be used to submit a Checkpoint, comprised of the properties on the Checkpoint class.
Returns:: The results of stream creation
Return type:: Checkpoint

get_checkpoint(stream_name, app_name, partition_id, checkpoint_type='LAST_READ')

Query details of a Checkpoint.

Parameters:

stream_name – Name of the stream to which the checkpoint belongs.
app_name – Name of the app associated with the checkpoint.
partition_id – Identifier of the stream partition.
checkpoint_type – Type of the checkpoint (Default: LAST_READ).

Returns:

One Checkpoint

Raises:

ResourceNotFound when no resource can be found.

delete_checkpoint(stream_name, app_name, partition_id=None, checkpoint_type='LAST_READ')

Delete a Checkpoint.

Parameters:

stream_name – Name of the stream to which the checkpoint belongs.
app_name – Name of the app associated with the checkpoint.
partition_id – Identifier of the stream partition (Default: None)
checkpoint_type – Type of the checkpoint (Default: LAST_READ).

Returns:

None

Data Operations¶

class otcextensions.sdk.dis.v2._proxy.Proxy(session, *args, **kwargs)

upload_data(stream_name, stream_id=None, records=None, filename=None)

Upload data to DIS stream.

Parameters:

stream_name – Name of the stream.
stream_id – Optional stream ID.
records – List of records (if filename is not used).
filename – Path to the CSV file.

Returns:

The results of uploaded data.

Return type:

Data

download_data(partititon_cursor, max_fetch_bytes=None, filename=None)

Download data from a DIS stream.

Parameters:

partititon_cursor – Data cursor, which needs to be obtained through the API for obtaining data cursors.
max_fetch_bytes – Maximum number of bytes that can be obtained for each request.
filename – Path to the CSV file.

Returns:

a generator of (Data) instances

get_data_cursor(stream_name, partition_id, **params)

Query data cursor.

Parameters:

stream_name – Name of the stream.
partititon_id – Partition ID of the stream.
params (dict) – Optional query parameters.

Returns:

partition_cursor.

Return type:

Data

Dump Task Operations¶

class otcextensions.sdk.dis.v2._proxy.Proxy(session, *args, **kwargs)

create_dump_task(stream_name, **attrs)

Add OBS dump tasks.

Parameters:

stream_name – Name of the stream.
attrs (dict) – Keyword arguments which will be used to create a DumpTask, comprised of the properties on the DumpTask class.

Returns:

The results of adding DumpTask.

Return type:

DumpTask

dump_tasks(stream_name)

List Dump tasks.

Parameters:: stream_name – Name of the stream.
Returns:: a generator of (DumpTask) instances

delete_dump_task(stream_name, task_name, ignore_missing=False)

Delete Dump Task.

Parameters:

stream_name – Name of the stream.
task_name – Name of the dump task to be deleted.

Returns:

None

get_dump_task(stream_name, task_name)

Query Dump Task details.

Parameters:

stream_name – Name of the stream..
task_name – Name of the dump task to be deleted.

Returns:

One DumpTask

Raises:

ResourceNotFound when no resource can be found.

start_dump_task(stream_name, *task_id)

Start dump tasks in batches.

Parameters:

stream_name – Name of the stream.
task_id – Dump task ID(s).

Returns:

None

pause_dump_task(stream_name, *task_id)

Pause dump tasks in batches.

Parameters:

stream_name – Name of the stream.
task_id – Dump task ID(s).

Returns:

None

Stream Operations¶

class otcextensions.sdk.dis.v2._proxy.Proxy(session, *args, **kwargs)

streams(**params)

List all Streams.

Parameters:: params (dict) – Optional query parameters to be sent to limit the streams being returned.
Returns:: a generator of (Stream) instances

get_stream(stream)

Query details of a DIS stream

Parameters:: stream – The value can be the name of a DIS Stream.
Returns:: One Stream
Raises:: ResourceNotFound when no resource can be found.

create_stream(**attrs)

Create a stream from attributes

Parameters:: attrs (dict) – Keyword arguments which will be used to create a Stream, comprised of the properties on the Streams class.
Returns:: The results of stream creation
Return type:: Stream

delete_stream(stream_name, ignore_missing=False)

Delete a DIS stream.

Parameters:

stream_name – Name of a DIS stream.
ignore_missing (bool) – When set to False ResourceNotFound will be raised when the stream does not exist. When set to True, no exception will be set when attempting to delete a nonexistent stream.

Returns:

None

update_stream_partition(stream_name, count)

Update a DIS stream partition count.

Parameters:

stream_name – Name of a DIS stream.
count – Number of the target partitions.

Returns:

None

last updated: 2024-05-18 00:15