Transferring Logs to OBS¶
You can transfer logs to OBS and download log files from the OBS console. You can choose scheduled or one-time transfers.
Note
To use this function, you must have the OBS Administrator permissions apart from the LTS permissions.
LTS transfers only logs generated after the transfer task is configured, not historical logs.
Prerequisites¶
Logs have been ingested to LTS. For details, see Log Ingestion.
You have created an OBS bucket.
Creating a Scheduled Log Transfer Task¶
Log in to the management console and choose Management & Deployment > Log Tank Service.
Choose Log Transfer in the navigation pane.
Click Configure Log Transfer in the upper right corner.
On the displayed page, configure the log transfer parameters.
Table 1 Transfer parameters¶ Parameter
Description
Example
Log Source Account
Current: Logs of the current account will be transferred.
Other: Logs of the delegator account will be transferred. Ensure that the delegator has created an agency for log transfer delegation. For details, see section "Creating an Agency".
Current
Transfer Mode
If you select the current account as the log source account, you can select Scheduled or One-time.
Scheduled: Logs are periodically transferred to OBS for long-term storage.
One-time: Logs are transferred to OBS for long-term storage in a one-time way.
Scheduled
Agency Name
This parameter is required when Log Source Account is set to Other. Enter the name of the agency created by the delegator.
-Delegator Account Name
This parameter is required when Log Source Account is set to Other. Enter the account name of the delegator.
-Enable Transfer
Enabled by default.
Enabled
Transfer Destination
Select a cloud service for log transfer.
OBS
Log Group Name
Select a log group.
N/A
Enterprise Project Name
Select an enterprise project.
This parameter is displayed only when the enterprise project function is enabled for the current account.
If the enterprise project function is enabled for the current account:
All enterprise projects under the current account are displayed in the drop-down list when Log Source Account is set to Current.
default is displayed when Log Source Account is set to Other and the enterprise project function is not enabled for the delegator account.
All enterprise projects under the delegator account are displayed when Log Source Account is set to Other and the enterprise project function is enabled for the delegator account.
-Log Stream Name
Select a log stream.
Note
Log streams that have been configured with OBS transfer settings cannot be configured again.
-OBS Bucket
Select an OBS bucket.
If no OBS buckets are available, click View OBS Bucket to access the OBS console and create an OBS bucket.
Currently, LTS supports only standard OBS buckets with the single-AZ storage policy.
Note
If you select an unauthorized OBS bucket, LTS will take 15 minutes to authorize the ACL for the bucket. If your configuration fails, try again 15 minutes later. To prevent log transfer failures, exercise caution when modifying bucket policies.
-Custom Log Transfer Path
Enabled: Logs will be transferred to a custom path to separate transferred log files of different log streams.
The format is /LogTanks/Region name/Custom path. The default custom path is lts/%Y/%m/%d, where %Y indicates the year, %m indicates the month, and %d indicates the day. A custom path must meet the following requirements:
Must start with /LogTanks/Region name.
Can contain only letters, digits, and the following special characters:
&$@;:,=+?-._/%. The character % can only be followed only by Y (year), m (month), d (day), H (hour), and M (minute). Any number of characters can be added before and after %Y, %m, %d, %H, and %M, and the sequence of these variables can be changed.Can contain 1-128 characters.
Example:
If you enter LTS-test/%Y/%m/%d/%H/%m, the path is LogTanks/Region name/LTS-test/Y/m/d/H/M/Log file name.
If you enter LTS-test/%d/%H/%m/%Y, the path is LogTanks/Region name/LTS-test/d/H/m/Y/Log file name.
Disabled: Logs will be transferred to the default path. The default path is LogTanks/Region name/2019/01/01/Log group/Log stream/Log file name.
LTS-test/%Y/%m/%d/%H/%M
Log Prefix
The file name prefix of the log files transferred to an OBS bucket
The prefix must meet the following requirements:
Can contain 0 to 64 characters.
Can contain only letters, digits, hyphens (-), underscores (_), and periods (.).
Example: If you enter LTS-log, the log file name will be LTS-log_Log file name.
LTS-log
Format
The storage format of logs. The value can be Raw Log Format or JSON.
Example of the raw log format:
(Logs displayed on the LTS console are in the raw format.)
Sep 30 07:30:01 ecs-bd70 CRON[3459]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1)
The following is an example of the JSON format:
{"host_name":"ecs-bd70","ip":"192.168.0.54","line_no":249,"message":"Sep 30 14:40:01 ecs-bd70 CRON[4363]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1)\n","path":"/var/log/syslog","time":1569825602303}
Json
Log Transfer Interval
The interval for automatically transferring logs to OBS buckets. The value can be 2, 5, or 30 minutes, or 1, 3, 6, or 12 hours.
3 hours
Time Zone
When logs are transferred to OBS buckets, the time in the transfer directory and file name will use the specified UTC time zone.
(UTC) Coordinated Universal Time
Filter by Tag Fields
During transfer, logs will be filtered by tag fields collected by ICAgent.
Disabled: Logs will not be filtered by tag fields.
Enabled: Default tag fields include those for hosts (hostIP, hostId, hostName, pathFile, and collectTime) and for Kubernetes (clusterName, clusterId, nameSpace, podName, containerName, and appName). Optional public tag fields are regionName, logStreamName, logGroupName, and projectId.
Note
When Filter by Tag Fields is enabled, Format must be JSON.
Filter by Tag Fields: When this parameter is enabled, logs will be filtered by tags.
Enabled
Compressed Format
Non-compression and gzip/zip compression are supported.
gzip
Click OK. When the log transfer status changes to Normal, the transfer task has been created.
Click the OBS bucket name in the Transfer Destination column to switch to the OBS console and view the transferred log files.
Transferred logs can be downloaded from OBS to your local computer for viewing.
Figure 1 Transferring logs to OBS¶
Note
Logs stored in OBS are in raw or JSON format.
Creating a One-off Log Transfer Task¶
Click Configure Log Transfer in the upper right corner.
On the displayed page, configure the log transfer parameters.
Table 2 Transfer parameters¶ Parameter
Description
Example
Transfer Mode
Scheduled: Logs are periodically transferred to OBS for long-term storage.
One-time: Logs are transferred to OBS for long-term storage in a one-time way.
One-time
Transfer Destination
Select a cloud service for log transfer.
OBS
Log Group Name
Select a log group.
N/A
Enterprise Project Name
Select an enterprise project.
This parameter is displayed only when the enterprise project function is enabled for the current account.
If the enterprise project function is enabled for the current account:
All enterprise projects under the current account are displayed in the drop-down list when Log Source Account is set to Current.
default is displayed when Log Source Account is set to Other and the enterprise project function is not enabled for the delegator account.
All enterprise projects under the delegator account are displayed when Log Source Account is set to Other and the enterprise project function is enabled for the delegator account.
-Log Stream Name
Select a log stream.
Note
Log streams that have been configured with OBS transfer settings cannot be configured again.
-Filter By
Keyword is selected by default. Enter the keyword to be filtered in the text box.
-Log Time Range
There are three types of time range: relative time from now, relative time from last, and specified time. Select a time range as required.
From now: queries log data generated in a time range that ends with the current time, such as the previous 1, 5, or 15 minutes. For example, if the current time is 19:20:31 and 1 hour is selected as the relative time from now, the charts on the dashboard display the log data that is generated from 18:20:31 to 19:20:31.
From last: queries log data generated in a time range that ends with the current time, such as the previous 1 or 15 minutes. For example, if the current time is 19:20:31 and 1 hour is selected as the relative time from last, the charts on the dashboard display the log data that is generated from 18:00:00 to 19:00:00.
Specified: queries log data that is generated in a specified time range.
-Total Log Events
Total number of log events.
-Log Files
Max log events for each transfer: 20 million. Max transfer files: 200.
-OBS Bucket
Select an OBS bucket.
If no OBS buckets are available, click View OBS Bucket to access the OBS console and create an OBS bucket.
Currently, LTS supports only Standard OBS buckets.
Data cannot be transferred to an OBS bucket whose storage class is Archive or for which cross-region replication has been configured.
-Bucket Directory
OBS bucket directory.
-Transfer File Name
Custom transfer file name. Only letters, digits, hyphens (-), underscores (_), and periods (.) are allowed.
-Format
Storage format of logs. The value can be Raw Log Format, JSON, or CSV.
Example of the raw log format:
(Logs displayed on the LTS console are in the raw format.)
Sep 30 07:30:01 ecs-bd70 CRON[3459]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1)
Example of the JSON format:
{"host_name":"ecs-bd70","ip":"192.168.0.54","line_no":249,"message":"Sep 30 14:40:01 ecs-bd70 CRON[4363]: (root) CMD (/opt/oss/servicemgr/ICAgent/bin/manual/mstart.sh > /dev/null 2>&1)\n","path":"/var/log/syslog","time":1569825602303}
CSV: Log content is displayed in a table.
Json
Click OK. When the log transfer status changes to Normal, the transfer task has been created.
Click the OBS bucket name in the Transfer Destination column to switch to the OBS console and view the transferred log files.
Download the transferred logs from OBS for viewing.
Modifying a Log Transfer Task¶
Locate the row that contains the target transfer task and click Modify in the Operation column.
Click OK.
Viewing Transfer Details¶
Locate the target log transfer task and click More > Details in the row of the desired task to view its details.
On the displayed Transfer Details page, you can view the log transfer details.
Deleting a Log Transfer Task¶
If logs do not need to be transferred, you can delete the transfer task.
Note
After a transfer task is deleted, log transfer will be stopped. Exercise caution when performing the deletion.
After a transfer task is deleted, the logs that have been transferred remain in OBS.
When you create a transfer task, OBS will grant read and write permissions to LTS for the selected bucket. If one OBS bucket is used by multiple transfer tasks, perform the following operations to delete the transfer task:
If only one transfer task is created using this OBS bucket, delete the bucket access permission granted to specific users on the Access Control > Bucket ACLs tab page on the OBS console when you delete the transfer task.
If multiple transfer tasks are created using this OBS bucket, do not delete the bucket access permission. Otherwise, data transfer will fail.
Locate the row of the target transfer task and choose Delete in the Operation column.
Click OK.
Viewing Transfer Status¶
The status of a transfer task can be Normal, Abnormal, or Disabled.
Normal: The log transfer task works properly.
Abnormal: An error occurred in the log transfer task. The possible cause is that the access control on the OBS bucket is configured incorrectly. Access the OBS console to correct the settings.
Disabled: The log transfer task is stopped.