Using LTS Search Syntax

LTS provides a range of search syntax to set search criteria and filtering rules for filtering records that meet the search criteria. Then, you can apply analysis statements on the filtering results for advanced analysis and processing.

To quickly understand and use the search syntax, you are advised to learn Search Modes, Phrase Search, Operator, and Search Statement Examples.

Note

  • Before using the search syntax, set the delimiters in Index Settings. If there is no special requirement, use the default delimiters: ,'";=()[]{}@&<>/:\\?\n\t\r and spaces.

  • The search syntax does not support search by delimiter.

    Search statements do not support delimiters. For example, in the search statement var/log, / is a delimiter. The search statement is equivalent to var log and is used to search for all logs that contain both var and log. Similarly, the search statements such as "var:log" and var;log are used to search for all logs that contain both var and log.

Search Modes

Search statements are used to define the filter criteria for log search and obtain the logs that meet the criteria.

Depending on the index configuration mode, it can be classified into full-text search and field search; according to the search accuracy, it can be classified into exact search and fuzzy search. Other types of search modes include range search and phrase search.

Table 1 Search modes

Search Mode

Description

Example

By full text

LTS splits an entire log into multiple keywords when full-text index is set.

Note

  • content is a built-in field corresponding to the original log text. The search statement GET is equivalent to content:GET. By default, the original log content is matched.

  • By default, multiple keywords are connected through AND. The search statement GET POST is equivalent to GET and POST.

  • GET POST

  • GET and POST

  • content:GET and content:POST

The preceding search statements have the same function, indicating that logs containing both GET and POST are searched.

By field

Search for specified field names and values (key:value) after field indexing is configured. You can perform multiple types of basic search and combined search based on the data type set in the field index.

Note

  • The value parameter cannot be empty. You can use the key:"" statement to search for logs with empty field values.

  • When field search is used together with the not operator, logs that do not contain this field are matched.

  • request_time>60 and request_method:po* indicates that the system searches for logs in which the value of request_time is greater than 60 and the value of request_method starts with po.

  • request_method:"" indicates that logs in which the value of request_method is empty are searched.

  • not request_method:GET indicates that logs that do not contain the request_method field and whose request_method value is not GET are searched.

By exact match

Use exact words for search.

LTS searches with word segmentation, which does not define the sequence of keywords.

Note

If the search statement is abc def, all logs that contain both abc and def are matched, for example, logs abc def and def abc. To ensure the sequence of keywords, use #"abc def".

  • GET POST indicates that logs containing both GET and POST are searched.

  • request_method:GET indicates that logs in which the value of request_method contains GET are searched.

  • #"/var/log" indicates that logs containing phrase /var/log are searched.

By fuzzy match

Specify a keyword in the search statement and add a wildcard, that is, an asterisk (*) or a question mark (?), to the middle or end of the keyword. LTS searches all logs for 100 words that meet the search criteria and returns logs that contain the words. The more precise the specified word is, the more accurate the search results are.

Note

  • The asterisk (*) indicates that multiple characters are matched, and the question mark (?) indicates that one character is matched.

  • Words cannot start with an asterisk (*) or a question mark (?).

  • Long and float data does not support fuzzy search using asterisks (*) or question marks (?).

  • If the fuzzy condition prefix is short and more than 100 words meet the criteria, the search results will be inaccurate.

  • GE* indicates that the system searches for words starting with GE in all logs and returns logs containing these words.

  • request_method:GE* indicates that the system searches for request_method values starting with GE in all logs and returns logs containing these words.

By scope

The long and float data supports range search.

  • Method 1: Use operators such as = (equal to) > (greater than) < (less than) operators to search for logs.

  • Method 2: Use the in operator to search for logs. The open/closed interval can be modified.

    Note

    The string fields do not support range query.

  • request_time>=60 indicates that the system searches for logs whose request_time value is greater than or equal to 60.

  • request_time in (60 120] indicates that the system searches for logs whose request_time value is greater than 60 and less than or equal to 120.

By phrase

Phrase search is used to fully match target phrases in logs to ensure the sequence in which keywords appear.

Note

Fuzzy search is not supported for phrase search.

#"abc def" indicates that the system searches all logs for the logs that contain the target phrase abc def.

  • Delimiters

    LTS splits the log content into multiple words based on delimiters. Default delimiters include ,'";=()[]{}@&<>/:\\?\n\t\r and spaces.

    For example, the default delimiters divide the log 2023-01-01 09:30:00 into four parts: 2023-01-01, 09, 30, and 00.

    In this case, the search statement 2023 cannot match the log. You can search for the log using 2023-01* or 2023-01-01.

    If the delimiter is set to null, the field value is regarded as a whole. You can search for the corresponding log only through complete log content or fuzzy search.

  • Keyword sequence

    Only the phrase search #"abc def" can ensure the sequence of keywords. In other search modes, multiple keywords are connected by AND.

    For example, request_method:GET POST is used to query logs that contain both GET and POST, and the sequence of GET and POST is not ensured. To ensure their sequence, Phrase Search is recommended.

  • Chinese search

    Fuzzy search is not required for Chinese search. Phrase search is recommended to match more accurate results.

    In LTS, English content is split into words of different lengths. Therefore, you can use fuzzy search to match logs with English words with the same prefix.

    Unigram segmentation is used to a Chinese string into Chinese characters. Each Chinese character is independent, and the length of each part is 1 character.

    For example, the search statement Monday indicates that logs containing M, o, n, d, a, and y are searched. The search statement #"Monday" indicates that logs containing the target phrase Monday are searched.

  • Invalid keyword

    The syntax keywords of log search statements include: && || AND OR and or NOT not in : > < = ( ) [ ]

    When and AND or OR NOT not in are used as syntax keywords, separate them with a space.

    If the log contains syntax keywords and needs to be searched, the search statement must be enclosed in double quotation marks. Otherwise, syntax errors may occur or incorrect results may be found.

    For example, if the search statement content:and contains the syntax keyword and, change it to content:"and".

Operator

For details about operators supported by the search statements, see Table 3.

Note

  • Except the in operator, other operators are case-insensitive.

  • The priorities of operators in descending order are as follows:

    1. Colon (:)

    2. Double quotation marks ("")

    3. Parentheses: ()

    4. and, not

    5. or

Table 3 Operators

Operator

Description

and

If there is no syntax keyword between multiple keywords, the and relationship is used by default. For example, GET 200 is equivalent to GET and 200.

Note

When and is used as an operator, use a space before and after it. For example, 1 and 2 indicates that logs containing both 1 and 2 are searched, and 1and2 indicates that logs containing 1and2 are searched.

AND

AND operator, equivalent to and.

&&

AND operator, for example, 1&&2.

Note

When && is used as an operator, spaces are not necessary. For example, 1 && 2 is equivalent to 1&&2, indicating that logs containing both 1 and 2 are searched.

or

Example: request_method:GET or status:200

Note

When or is used as an operator, use a space before and after it.

OR

OR operator, equivalent to or.

||

OR operator. When || is used as an operator, spaces are not necessary.

not

NOT operator. Example: request_method:GET not status:200, not status:200

Note

  • When not is used as an operator, use a space before and after it.

  • When field search is used together with the not operator, logs that do not contain this field are matched.

( )

Specify fields that should be matched with higher priority. Example: (request_method:GET or request_method:POST) and status:200

:

Search for a specified field (key:value), for example, request_method:GET.

Note

Use double quotation marks ("") to enclose a field name (key) or value that contains reserved characters, such as spaces and colons (:). Examples:

  • "request method":GET

  • message:"This is a log"

  • time:"09:00:00"

  • ipv6:"2024:AC8:2ac::d09"

""

Enclose a syntax keyword to convert it into common characters. For example, "and" means searching for logs that contain this word. The word and here is not an operator.

\

Escape double quotation marks (""). The escaped quotation marks indicate the symbol itself. For example, to search for instance_id:nginx"01", use instance_id:nginx\"01\".

*

An asterisk is a wildcard that matches zero, single, or multiple characters. Example: request_method:P*T

Note

Put * in the middle or at the end of a keyword.

?

A question mark matches a single character. For example, request_method:P?T can match PUT but cannot match POST.

Note

Put ? in the middle or at the end of a keyword.

>

Searches logs in which the value of a field is greater than a specified value. Example: request_time>100

>=

Searches logs in which the value of a field is greater than or equal to a specified value. Example: request_time>=100

<

Searches logs in which the value of a field is less than a specified value. Example: request_time<100

<=

Searches logs in which the value of a field is less than or equal to a specified value. Example: request_time<=100

=

Searches logs in which the value of a field is equal to a specified value, applying only to float or long fields. For fields of this type, the equal sign (=) and colon (:) have the same function. For example, request_time=100 is equivalent to request_time:100.

in

Search logs whose field values are in a specified range. Brackets indicate a closed interval, and parentheses indicate an open interval. Numbers are separated with spaces. Examples: request_time in [100 200] and request_time in (100 200]

Note

Enter in in lowercase. When it is used as an operator, use a space before and after it.

#""

Searches for logs that contain the target phrase, ensuring the sequence of keywords.

Note

The asterisk (*) and question mark (?) in phrase search are regarded as common characters. Therefore, phrase search does not support fuzzy search and can be used to search for the asterisk (*) and question mark (?) in logs.

Search Statement Examples

For the same search statement, different search results are displayed for different log content and index configurations. This section describes search statement examples based on the following log samples and indexes.

**Figure 1** Search example

Figure 1 Search example

Table 4 Search statement examples

Search Requirement

Search Statement

Logs of POST requests whose status code is 200

request_method:POST and status=200

Logs of successful GET or POST requests (status codes 200 to 299)

(request_method:POST or request_method:GET) and status in [200 299]

Logs of failed GET or POST requests

(request_method:POST or request_method:GET) not status in [200 299]

Logs of non-GET requests

not request_method:GET

Logs of successful GET request and request time is less than 60 seconds

request_method:GET and status in [200 299] not request_time>=60

Logs whose request time is 60 seconds.

  • request_time:60

  • request_time=60

Logs of requests whose time is greater than or equal to 60 seconds and less than 200 seconds

  • request_time>=60 and request_time<200

  • request_time in [60 200)

Logs that contain and

content:"and"

Note

Double quotation marks are used to enclose and. In this case, and is a common string, not an operator.

Logs that do not contain the user field.

not user:*

Logs in which the value of user is empty are searched.

user:""

Logs in which the value of the week field is not Monday

not week: Monday

Logs in which the value of sec-ch-ua-mobile is ?0 are searched.

sec-ch-ua-mobile:#"?0"

Note

If search is required when log content contains asterisks (*) or question marks (?), use phrases search.

For more complex search examples, see Table 5.

Table 5 Fuzzy search

Search Requirement

Search Statement

Logs that contain words starting with GE

GE*

Logs that contain words starting with GE and with only one character after GE.

GE?

Logs in which the value of request_method contains a word starting with G.

request_method:G*

Logs in which the value of request_method starts with P, ends with T, and contains a single character in the middle.

request_method:P?T

Logs in which the value of request_method starts with P, ends with T, and contains zero, single, or multiple characters in the middle.

request_method:P*T

Search based on delimiters. For example, the value of the User-Agent field is Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36.

  • If this parameter is left blank, the value of this field is considered as a whole. In this case, when you use User-Agent:Chrome to search for logs, no log can be found.

  • When the delimiter is set to , '";=()[]{}?@&<>/:\n\t\r, the value of this field is split into Mozilla, 5.0, Windows, NT, 10.0, Win64, x64, AppleWebKit, 537.36, KHTML, like, Gecko, Chrome, 113.0.0.0, Safari, and 537.36.

    Then you can use search statements such as User-Agent:Chrome for search.

    Table 6 Delimiter-based search

    Search Requirement

    Search Statement

    Logs in which the value of User-Agent contains Chrome

    User-Agent:Chrome

    Logs in which the value of User-Agent contains the word starting with Win

    User-Agent:Win*

    Logs in which the value of User-Agent contains Chrome and Linux

    User-Agent:"Chrome Linux"

    Logs in which the value of User-Agent contains Firefox or Chrome

    User-Agent:Chrome OR User-Agent:Linux

    Logs in which the value of User-Agent contains Chrome but not Linux

    User-Agent:Chrome NOT User-Agent:Linux

last updated: 2024-11-14 08:18 UTC - commit: 1acdc5d6d982953478f58e5ac5d5a8d5e2ed9081