Developing a Python Script

This section describes how to develop and execute a Python script using DataArts Factory.

Preparing the Environment

• An ECS named ecs-dgc has been created.

  Note

  In this example, the ECS uses the CentOS 8.0 64bit with ARM (40 GB) public image and the Python environment. You can log in to the ECS and run the python command to check the Python environment.

  

• You have enabled the DataArts Migration incremental package and created a CDM cluster named cdm-dlfpyhthon. The cluster provides an agent for the DataArts Factory module to communicate with the ECS.

• Ensure that the ECS can communicate with the CDM cluster, which depends on the following conditions:

  • If the CDM cluster and the ECS are in the same region, VPC, subnet, and security group, they can communicate with each other by default. If they are in the same VPC but in different subnets or security groups, you must configure routing rules and security group rules. For details about how to configure routing rules, see Adding Routes in Virtual Private Cloud (VPC) Usage Guide. For details about how to configure security group rules, see Security Group > Adding a Security Group Rule in Virtual Private Cloud (VPC) Usage Guide.

  • If the CDM cluster and the ECS are in different regions, a public network or a dedicated connection is required for enabling communication between the CDM cluster and the cloud service. If the Internet is used for communication, ensure that an EIP has been bound to the CDM cluster, the host where the data source is located can access the Internet, and the port has been enabled in the firewall rules.

  • The ECS and the CDM cluster belong to the same enterprise project. If they do not, you can modify the enterprise project of the workspace.

Constraints

• Python scripts do not support script parameters or job parameters.

Creating an ECS Data Connection

Before developing a Python script, you need to create a connection to the ECS.

1. On the DataArts Studio console, locate a workspace and click Management Center.

   

   Figure 1 Management Center

2. In the navigation pane, choose Manage Data Connections.

   

   Figure 2 Manage Data Connections

3. Click Create Data Connection.

   

   Figure 3 Create Data Connection

4. Configure parameters by referring to Table 1 and create a data connection named python_test.

   Table 1 Host Connection

   Parameter	Mandatory	Description
   Data Connection Name	Yes	Name of the host connection. The value can contain only letters, digits, hyphens (-), and underscores (_).
   Host Address	Yes	IP address of the host For details, see section "Viewing Details About an ECS" in Elastic Cloud Server User Guide.
   Agent	Yes	Agents provided by the CDM cluster.
   Port	Yes	SSH port number of the host
   Username	Yes	Username of the host
   Login Mode	Yes	Mode for logging in to the host Key pair Password
   Key Pair	Yes	If you select Key pair for Login Mode, you need to obtain the private key file, upload it to OBS, and select the OBS path. This parameter is available only when Login Mode is set to Key pair. Note The uploaded private key file must be in PEM format, and the uploaded private key file and the public key configured on the host must be in the same key pair.
   Key Pair Password	No	If no password is set for the key pair, you do not need to set this parameter.
   Password	Yes	Password for logging in to the host.
   Host Connection Description	No	Description of the host connection

   Note

   The key parameters are as follows:

   • Host Address: Enter the IP address of the ECS.

   • Agent: Select the CDM cluster.

5. Click Test to test connectivity of the data connection. If the test passes, the data connection is created.

6. After the test is successful, click OK. The system will create the data connection for you.

Developing a Python Script

1. Choose DataArts Factory > Develop Script and create a Python script named python_test.

2. Edit the Python statement in the editor, select the host connection, and click Submit and Unlock..

   Note

   • This example defines a string template for saving company information and uses the template to output information about different companies.

     import sys
     Company_Name1=sys.argv[1]
     Company_Name2=sys.argv[2]
     template='No.:{:0>9s} \t CompanyName: {:s} \t Website: https://www.{:s}.com'
     context1=template.format('1',Company_Name1,Company_Name1.lower())
     context2=template.format('2',Company_Name2,Company_Name2.lower())
     print(context1)
     print(context2)
     

   • The script development area in Figure 4 is a temporary debugging area. After you close the script tab, the development area will be cleared.

   • Connection: Select the data connection created in Creating an ECS Data Connection.

   

   Figure 4 Editing the Python statement

3. Click Execute to execute the Python statement.

4. View the script execution result.

last updated: 2024-07-03 00:13