Table 1 Job parameters when Elasticsearch or CSS is the source¶
Parameter
Description
Example Value
Index
Elasticsearch index, which is similar to the name of a relational database. The index name can contain only lowercase letters.
index
Type
Elasticsearch type, which is similar to the table name of a relational database. The type name can contain only lowercase letters.
type
Split Nested Field
(Optional) Whether to split the JSON content of the nested fields. For example, a:{ b:{ c:1, d:{ e:2, f:3 } } } can be split into a.b.c, a.b.d.e, and a.b.d.f.
No
Filter Conditions
(Optional) CDM migrates only the data that meets the filter conditions.
Currently, only the query string (q syntax) of Elasticsearch can be used to filter source data. The q syntax is used in the following way:
In exact match, the column:data format is used to match and filter data. column indicates the field name, and data indicates the query condition, for example, last_name:Smith.
In addition, if data is a string containing spaces, it must be enclosed in double quotation marks. If column is not specified, all fields will be matched by data.
Multiple query conditions can be combined with connection words. The format is column1:data1ANDcolumn2:data2. The connection words can be AND, OR, or NOT. They must be in uppercase, and there must be a space before and after each connection word.
Example: last_name:Smith AND last_name:John
In range matching, you can directly use a condition expression to filter data. The expression is in column:>data format. The operator can be >, >=, <, or <=.
An example is time:>=1636905600000 AND time:<1637078400000. It can also be used together with a macro variable of date and time, for example, createTime:>=${timestamp(dateformat(yyyyMMdd,-1,DAY))} AND createTime:< ${timestamp(dateformat(yyyyMMdd))}.
In range matching, you can also use the range syntax to filter data. The format is column:{data1 TO data2}. { and } indicate that a value is not included. [ and ] indicate that a value is included. TO must be capitalized, and there must be a space before and after it. * indicates all data.
For example, time:{1636992000000 TO *] filters out all the data greater than 1636992000000 in the time field. It can also be used together with a macro variable of date and time, for example, createTime:[${timestamp(dateformat(yyyyMMdd,-1,DAY))} TO ${timestamp(dateformat(yyyyMMdd))}}.
Source data cannot be filtered using the query domain-specific language (DSL) of Elasticsearch.
last_name:Smith
Extract Meta-field
Whether to extract index meta-fields. For example, _index, _type, _id, and _score.
Yes
On the Map Field page, you can set custom fields for the source and destination.
