Creating a Smart Connect Task¶

Function¶

This API is used to create a Smart Connect task.

URI¶

POST /v2/{project_id}/instances/{instance_id}/connector/tasks

**Table 1** Path Parameters¶
Parameter	Mandatory	Type	Description
project_id	Yes	String	Project ID. For details, see Obtaining a Project ID.
instance_id	Yes	String	Instance ID.

Request Parameters¶

**Table 2** Request body parameters¶
Parameter	Mandatory	Type	Description
task_name	No	String	Smart Connect task name.
start_later	No	Boolean	Indicates whether to start a task later. false: to create a task and start immediately; true: to create a task and manually start it later in the task list.
topics	No	String	Topic of a Smart Connect task.
topics_regex	No	String	Regular expression of the topic of a Smart Connect task.
source_type	No	String	Source type of a Smart Connect task.
source_task	No	SmartConnectTaskReqSourceConfig object	Source configuration of a Smart Connect task.
sink_type	No	String	Target type of a Smart Connect task.
sink_task	No	SmartConnectTaskReqSinkConfig object	Target type of a Smart Connect task.

**Table 3** SmartConnectTaskReqSourceConfig¶
Parameter	Mandatory	Type	Description
redis_address	No	String	Redis instance address. (Mandatory only when the source type is Redis.)
redis_type	No	String	Redis instance type. (Mandatory only when the source type is Redis.)
dcs_instance_id	No	String	DCS instance ID. (Mandatory only when the source type is Redis.)
redis_password	No	String	Redis password. (Mandatory only when the source type is Redis.)
sync_mode	No	String	Synchronization type: RDB_ONLY indicates full synchronization; CUSTOM_OFFSET indicates full and incremental synchronization. (Mandatory only when the source type is Redis.)
full_sync_wait_ms	No	Integer	Interval of full synchronization retries, in ms. (Mandatory only when the source type is Redis.)
full_sync_max_retry	No	Integer	Max. retries of full synchronization. (Mandatory only when the source type is Redis.)
ratelimit	No	Integer	Rate limit, in KB/s. -1: There is no rate limit. (Mandatory only when the source type is Redis.)
current_cluster_name	No	String	Current Kafka instance name. (Mandatory only when the source type is Kafka.)
cluster_name	No	String	Target Kafka instance name. (Mandatory only when the source type is Kafka.)
user_name	No	String	Username set when SASL_SSL was enabled for a target Kafka instance, or when a SASL_SSL user was created. (Mandatory only when the source type is Kafka and the target Kafka authentication mode is SASL_SSL.)
password	No	String	Password set when SASL_SSL was enabled for a target Kafka instance, or when a SASL_SSL user was created. (Mandatory only when the source type is Kafka and the target Kafka authentication mode is SASL_SSL.)
sasl_mechanism	No	String	Target Kafka authentication mode. (Mandatory only when the source type is Kafka and the authentication mode is SASL_SSL.)
instance_id	No	String	Target Kafka instance ID. (Mandatory only when the source type is Kafka. Specify either instance_id or bootstrap_servers.)
bootstrap_servers	No	String	Target Kafka instance address. (Mandatory only when the source type is Kafka. Specify either instance_id or bootstrap_servers.)
security_protocol	No	String	Target Kafka authentication. (Mandatory only when the source type is Kafka). There are two authentication modes: SASL_SSL: SASL_SSL is enabled. PLAINTEXT: SASL_SSL is disabled.
direction	No	String	Sync direction: pull replicates data from the target Kafka instance to the current one; push replicates data from the source Kafka instance to the target one; two-way replicates data from the source and target Kafka instance data to each other. (Mandatory only when the source type is Kafka.)
sync_consumer_offsets_enabled	No	Boolean	Indicates whether to sync the consumption progress. (Mandatory only when the source type is Kafka.)
replication_factor	No	Integer	Number of topic replicas when a topic is automatically created in the peer instance. The value of this parameter cannot exceed the number of brokers in the peer instance. This parameter overrides the default.replication.factor parameter configured in the target instance. (Mandatory only when the source type is Kafka.)
task_num	No	Integer	Number of data replication tasks. The default value is 2. You are advised to use the default value. If the sync direction is set to two-way, the actual number of tasks will be twice the number of tasks you configure here. (Mandatory only when the source type is Kafka.)
rename_topic_enabled	No	Boolean	Indicates whether to rename a topic. If yes, add the alias of the source Kafka instance before the target topic name to form a new name of the target topic. (Mandatory only when the source type is Kafka.)
provenance_header_enabled	No	Boolean	The target topic receives the replicated messages. The message header contains the message source. If you select Both for Sync Direction, enable Add Source Header to prevent infinite replication. (Mandatory only when the source type is Kafka.)
consumer_strategy	No	String	Start offset. latest: Obtain the latest data; earliest: Obtain the earliest data. (Mandatory only when the source type is Kafka.)
compression_type	No	String	Compression algorithm to use for copying messages. (Mandatory only when the source type is Kafka.) none gzip snappy lz4 zstd
topics_mapping	No	String	Topic mapping, which is used to customize the target topic name. Rename Topic and Topic Mapping cannot be configured at the same time. Topic mapping format: source topic:target topic. Use commas (,) to separate multiple topic mappings, for example, topic-sc-1:topic-sc-2,topic-sc-3:topic-sc-4. (Mandatory only when the source type is Kafka.)

**Table 4** SmartConnectTaskReqSinkConfig¶
Parameter	Mandatory	Type	Description
redis_address	No	String	Redis instance address. (Mandatory only when the target type is Redis.)
redis_type	No	String	Redis instance type. (Mandatory only when the target type is Redis.)
dcs_instance_id	No	String	DCS instance ID. (Mandatory only when the target type is Redis.)
redis_password	No	String	Redis password. (Mandatory only when the target type is Redis.)
consumer_strategy	No	String	Start offset. latest: Obtain the latest data; earliest: Obtain the earliest data. (Mandatory only when the target type is OBS.)
destination_file_type	No	String	Dump file format. Only TEXT is supported. (Mandatory only when the target type is OBS.)
deliver_time_interval	No	Integer	Data dumping intervals, in seconds. The default interval is 300s. (Mandatory only when the target type is OBS.)
access_key	No	String	AK: access key ID. (Mandatory only when the target type is OBS.)
secret_key	No	String	SK: secret access key used together with the access key ID. (Mandatory only when the target type is OBS.)
obs_bucket_name	No	String	Dumping address, which is the OBS bucket used to store the topic data. (Mandatory only when the target type is OBS.)
obs_path	No	String	Dumping directory, which is the directory for storing topic files dumped to OBS. Use slashes (/) to separate directory levels. (Mandatory only when the target type is OBS.)
partition_format	No	String	Time directory format. (Mandatory only when the target type is OBS.) yyyy: year. yyyy/MM: year and month. yyyy/MM/dd: year, month, and day. yyyy/MM/dd/HH: year, month, day, and hour. yyyy/MM/dd/HH/mm: year, month, day, hour, and minute.
record_delimiter	No	String	Line break, which is used to separate the user data that is written into the dump file. (Mandatory only when the target type is OBS.) Value range: Comma (,) Semicolon (;) Vertical bar (\|) Newline (\n) NULL
store_keys	No	Boolean	Specifies whether to dump keys. (Mandatory only when the target type is OBS.)

Response Parameters¶

Status code: 200

**Table 5** Response body parameters¶
Parameter	Type	Description
task_name	String	Smart Connect task name.
topics	String	Topic of a Smart Connect task.
topics_regex	String	Regular expression of the topic of a Smart Connect task.
source_type	String	Source type of a Smart Connect task.
source_task	SmartConnectTaskRespSourceConfig object	Source configuration of a Smart Connect task.
sink_type	String	Target type of a Smart Connect task.
sink_task	SmartConnectTaskRespSinkConfig object	Target type of a Smart Connect task.
id	String	ID of a Smart Connect task.
status	String	Smart Connect task status.
create_time	Long	Time when the Smart Connect task was created.

**Table 6** SmartConnectTaskRespSourceConfig¶
Parameter	Type	Description
redis_address	String	Redis instance address. (Displayed only when the source type is Redis.)
redis_type	String	Redis instance type. (Displayed only when the source type is Redis.)
dcs_instance_id	String	DCS instance ID. (Displayed only when the source type is Redis.)
sync_mode	String	Synchronization type: RDB_ONLY indicates full synchronization; CUSTOM_OFFSET indicates full and incremental synchronization. (Displayed only when the source type is Redis.)
full_sync_wait_ms	Integer	Interval of full synchronization retries, in ms. (Displayed only when the source type is Redis.)
full_sync_max_retry	Integer	Max. retries of full synchronization. (Displayed only when the source type is Redis.)
ratelimit	Integer	Rate limit, in KB/s. -1: disable. (Displayed only when the source type is Redis.)
current_cluster_name	String	Current Kafka instance name. (Displayed only when the source type is Kafka.)
cluster_name	String	Target Kafka instance name. (Displayed only when the source type is Kafka.)
user_name	String	Username of the target Kafka instance. (Displayed only when the source type is Kafka.)
sasl_mechanism	String	Target Kafka authentication mode. (Displayed only when the source type is Kafka.)
instance_id	String	Target Kafka instance ID. (Displayed only when the source type is Kafka.)
bootstrap_servers	String	Target Kafka instance address. (Displayed only when the source type is Kafka.)
security_protocol	String	Target Kafka authentication. (Displayed only when the source type is Kafka.)
direction	String	Sync direction. (Displayed only when the source type is Kafka.)
sync_consumer_offsets_enabled	Boolean	Indicates whether to sync the consumption progress. (Displayed only when the source type is Kafka.)
replication_factor	Integer	Number of replicas. (Displayed only when the source type is Kafka.)
task_num	Integer	Number of tasks. (Displayed only when the source type is Kafka.)
rename_topic_enabled	Boolean	Indicates whether to rename a topic. (Displayed only when the source type is Kafka.)
provenance_header_enabled	Boolean	Indicates whether to add the source header. (Displayed only when the source type is Kafka.)
consumer_strategy	String	Start offset. latest: Obtain the latest data; earliest: Obtain the earliest data. (Displayed only when the source type is Kafka.)
compression_type	String	Compression algorithm. (Displayed only when the source type is Kafka.)
topics_mapping	String	Topic mapping. (Displayed only when the source type is Kafka.)

**Table 7** SmartConnectTaskRespSinkConfig¶
Parameter	Type	Description
redis_address	String	Redis instance address. (Displayed only when the target type is Redis.)
redis_type	String	Redis instance type. (Displayed only when the target type is Redis.)
dcs_instance_id	String	DCS instance ID. (Displayed only when the target type is Redis.)
target_db	Integer	Target database. The default value is -1. (Displayed only when the target type is Redis.)
consumer_strategy	String	Start offset. latest: Obtain the latest data; earliest: Obtain the earliest data. (Displayed only when the target type is OBS.)
destination_file_type	String	Dump file format. Only TEXT is supported. (Displayed only when the target type is OBS.)
deliver_time_interval	Integer	Dumping period (s). (Displayed only when the target type is OBS.)
obs_bucket_name	String	Dumping address. (Displayed only when the target type is OBS.)
obs_path	String	Dump directory. (Displayed only when the target type is OBS.)
partition_format	String	Time directory format. (Displayed only when the target type is OBS.)
record_delimiter	String	Line break. (Displayed only when the target type is OBS.)
store_keys	Boolean	Storage key. (Displayed only when the target type is OBS.)
obs_part_size	Integer	Size (in bytes) of each file to be uploaded. The default value is 5242880. (Displayed only when the target type is OBS.)
flush_size	Integer	flush_size. (Displayed only when the target type is OBS.)
timezone	String	Time zone. (Displayed only when the target type is OBS.)
schema_generator_class	String	schema_generator class. The default value is io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator. (Displayed only when the target type is OBS.)
partitioner_class	String	partitioner class. The default value is io.confluent.connect.storage.partitioner.TimeBasedPartitioner. (Displayed only when the target type is OBS.)
value_converter	String	value_converter. The default value is org.apache.kafka.connect.converters.ByteArrayConverter. (Displayed only when the target type is OBS.)
key_converter	String	key_converter. The default value is org.apache.kafka.connect.converters.ByteArrayConverter. (Displayed only when the target type is OBS.)
kv_delimiter	String	kv_delimiter. The default value is :. (Displayed only when the target type is OBS.)

Example Requests¶

Creating a dumping task that starts immediately.

POST https://{endpoint}/v2/{project_id}/instances/{instance_id}/connector/tasks

{
  "task_name" : "smart-connect-1",
  "start_later" : false,
  "source_type" : "NONE",
  "topics_regex" : "topic-obs*",
  "sink_type" : "OBS_SINK",
  "sink_task" : {
    "consumer_strategy" : "earliest",
    "destination_file_type" : "TEXT",
    "deliver_time_interval" : 300,
    "access_key" : "********",
    "secret_key" : "********",
    "obs_bucket_name" : "obs_bucket",
    "obs_path" : "obsTransfer-1810125534",
    "partition_format" : "yyyy/MM/dd/HH/mm",
    "record_delimiter" : "\\n",
    "store_keys" : false
  }
}

Creating a Kafka data replication task that starts later.

POST https://{endpoint}/v2/{project_id}/instances/{instance_id}/connector/tasks

{
  "task_name" : "smart-connect-2",
  "start_later" : true,
  "source_type" : "KAFKA_REPLICATOR_SOURCE",
  "source_task" : {
    "current_cluster_name" : "A",
    "cluster_name" : "B",
    "user_name" : "user1",
    "password" : "********",
    "sasl_mechanism" : "SCRAM-SHA-512",
    "instance_id" : "b54c9dd8-********-********",
    "direction" : "two-way",
    "sync_consumer_offsets_enabled" : false,
    "replication_factor" : 3,
    "task_num" : 2,
    "rename_topic_enabled" : false,
    "provenance_header_enabled" : true,
    "consumer_strategy" : "latest",
    "compression_type" : "snappy",
    "topics_mapping" : "topic-sc-1:topic-sc-3,topic-sc-2:topic-sc-4"
  }
}

Creating a Redis data replication task that starts immediately: full synchronization, 10 maximum retry times, 10,000 ms retry interval, and 10 KB/s bandwidth limit.

POST https://{endpoint}/v2/{project_id}/instances/{instance_id}/connector/tasks

{
  "task_name" : "smart-connect-3",
  "start_later" : false,
  "source_type" : "REDIS_REPLICATOR_SOURCE",
  "source_task" : {
    "redis_address" : "192.168.99.249:6379,192.168.120.127:6379,192.168.116.219:6379",
    "redis_type" : "cluster",
    "redis_password" : "********",
    "sync_mode" : "RDB_ONLY",
    "full_sync_max_retry" : 10,
    "full_sync_wait_ms" : 10000,
    "ratelimit" : 10
  },
  "topics" : "topic-sc-3",
  "sink_type" : "REDIS_REPLICATOR_SINK",
  "sink_task" : {
    "redis_address" : "192.168.119.51:6379",
    "redis_type" : "standalone",
    "redis_password" : "********"
  }
}

Example Responses¶

Status code: 200

Successful.

{
  "task_name" : "smart-connect-121248117",
  "topics" : "topic-sc",
  "source_task" : {
    "redis_address" : "192.168.91.179:6379",
    "redis_type" : "standalone",
    "dcs_instance_id" : "949190a2-598a-4afd-99a8-dad3cae1e7cd",
    "sync_mode" : "RDB_ONLY,",
    "full_sync_wait_ms" : 13000,
    "full_sync_max_retry" : 4,
    "ratelimit" : -1
  },
  "source_type" : "REDIS_REPLICATOR_SOURCE",
  "sink_task" : {
    "redis_address" : "192.168.119.51:6379",
    "redis_type" : "standalone",
    "dcs_instance_id" : "9b981368-a8e3-416a-87d9-1581a968b41b",
    "target_db" : -1
  },
  "sink_type" : "REDIS_REPLICATOR_SINK",
  "id" : "8a205bbd-7181-4b5e-9bd6-37274ce84577",
  "status" : "RUNNING",
  "create_time" : 1708427753133
}

Status Codes¶

Status Code	Description
200	Successful.

Error Codes¶

See Error Codes.

last updated: 2024-11-13 14:47 UTC - commit: bab3492e4e325ec745b7b7fa1576930332ed8243