Public | DataArts Studio is a one-stop platform that provides data integration, development, and governance capabilities. DataArts Studio has no storage or computing capability and relies on the data lake base. Only one DataArts Studio instance can be bound to an enterprise project. If an enterprise project already has an instance, no more instance can be added. Different modules of DataArts Studio support different data sources. You need to select a data lake foundation based on your service requirements. For details about the data lakes supported by DataArts Studio, see "Management Center" > "Data Sources Supported by DataArts Studio" in DataArts Studio User Guide.
|
Management Center | Due to the constraints of Management Center, other modules (such as DataArts Architecture, DataArts Quality, and DataArts Catalog) do not support databases or tables whose names contain Chinese characters or periods (.). You are advised to use different CDM clusters for a data connection agent in Management Center and a CDM migration job. If an agent and CDM job use the same cluster, they may contend for resources during peak hours, resulting in service unavailability. If you use the same CDM cluster as the agent for multiple connections to MRS clusters with Kerberos authentication enabled, jobs will fail. You are advised to plan multiple CDM clusters based on service requirements. If a CDM cluster functions as the agent for a data connection in Management Center, the cluster supports a maximum of 200 concurrent active threads. If multiple data connections share an agent, a maximum of 200 SQL, Shell, and Python scripts submitted through the connections can run concurrently. Excess tasks will be queued. You are advised to plan multiple agents based on the workload. A maximum of 200 data connections can be created in a workspace. The concurrency restriction for APIs in Management Center is 100 QPS.
|
DataArts Migration | You can enable automatic backup and restoration of CDM jobs. Backups of CDM jobs are stored in OBS buckets. For details, see DataArts Migration > Job Management > Job Configuration Management in DataArts Studio User Guide. There is no quota limit for CDM jobs. However, it is recommended that the number of jobs be less than or equal to twice the number of vCPUs in the CDM cluster. Otherwise, job performance may be affected. The DataArts Migration cluster is deployed in standalone mode. A cluster fault may cause service and data loss. You are advised to use the CDM Job node of DataArts Factory to invoke CDM jobs and select two CDM clusters to improve reliability. For details, see DataArts Factory > Nodes > CDM Job in DataArts Studio User Guide. If changes occur in the connected data source (for example, the MRS cluster capacity is expanded), you need to edit and save the connection. If you have uploaded an updated version of a driver, you must restart the CDM cluster for the new driver to take effect. The number of concurrent extraction tasks for a job ranges from 1 to 300, and the total number of concurrent extraction tasks for a cluster ranges from 1 to 1,000. The maximum number of concurrent extraction tasks for a cluster depends on the CDM cluster specifications. You are advised to set the maximum number of concurrent extraction tasks to no larger than twice the number of vCPUs. The number of concurrent extraction tasks for a job should not exceed that for a cluster. If the number of concurrent extraction tasks is too large, memory overflow may occur. Exercise caution when changing the maximum number of concurrent extraction tasks.
For more constraints on DataArts Migration, see "DataArts Migration" > "Constraints" in DataArts Studio User Guide. |
DataArts Factory | You can enable backup of assets such as scripts and jobs to OBS buckets. For details, see DataArts Factory > O&M and Scheduling > Managing Backups in DataArts Studio User Guide. The execution history of scripts, jobs, and nodes is stored in OBS buckets. If no OBS bucket is available, you cannot view the execution history. Resources from an HDFS can be used only by MRS Spark, MRS Flink Job, and MRS MapReduce nodes. A workspace can contain a maximum of 10,000 scripts (including SQL, Shell, and Python scripts), 5,000 script directories, and 10 directory levels. A workspace can contain a maximum of 10,000 jobs (including batch processing jobs, real-time processing jobs, offline migration jobs, and real-time migration jobs), 5,000 job directories, and 10 directory levels. A workspace can contain a maximum of 500 real-time jobs (including real-time processing jobs and real-time migration jobs). A maximum of 1,000 execution results can be displayed for RDS SQL, DWS SQL, Hive SQL, DLI SQL, and Spark SQL scripts, and the data volume is less than 3 MB. If the number of execution results exceeds 1,000, you can dump them. A maximum of 10,000 execution results or 3 MB data can be dumped. Only data of the last six months can be displayed on the Monitor Instance and Monitor PatchData pages. Only notification records of the last 30 days can be displayed. The download records age out every seven days. When aged out, download records and the data dumped to OBS are both deleted.
|
DataArts Architecture | DataArts Architecture supports ER modeling, dimensional modeling (only star models), and data mart. The maximum size of a file to be imported is 4 MB. A maximum of 3,000 metrics can be imported. A maximum of 500 tables can be exported at a time. A maximum of 200 objects can be operated at a time in DataArts Architecture. The quotas for the objects in a workspace are as follows: Subjects: 5,000 Data standard directories: 500; data standards: 20,000 Business metrics: 100,000 Atomic, derivative, and compound metrics: 5,000 for each
The quotas for different custom objects are as follows: The name of a physical model table can contain a maximum of 200 characters. If you select Description, field comments of a table must be unique. If DataArts Architecture uses MRS Spark connections and DataArts Catalog uses MRS Hive connections to collect data, and if there are multiple Hive connections in the same cluster, you must use all these Hive connections to collect data in DataArts Catalog. Otherwise, tables may not be found during reversing in DataArts Architecture.
|
DataArts Quality | The execution duration of quality jobs and comparison jobs depends on the data engine. If the data engine does not have sufficient resources, the execution of data quality jobs may be slow. A maximum of 50 subjobs (rules) can be configured for a quality job. If necessary, you can create multiple quality jobs. A maximum of five subjobs (rules) can be configured for a comparison job. If necessary, you can create multiple comparison jobs. Each subjob of a quality job or comparison job can have a maximum of 300 data objects (databases, tables, and fields). By default, a maximum of 1,000 SQL statements associated with a quality job of a data connection can be executed concurrently. Excess SQL statements will be queued. The value ranges from 10 to 1000. By default, a maximum of 10,000 SQL statements associated with a quality job in a region can be executed concurrently. Excess SQL statements will be queued. In the Quantity Changes area on the Dashboard page on the Quality Monitoring page, data of 30 days can be displayed. In the Alarm Trend by Severity and Rule Quantity Trend areas, data of the last seven days can be displayed. Quality reports are generated in batches on the T+1 day and retained for 90 days. If you export a quality report to OBS, the report is exported to the OBS path for storing job logs configured for the workspace. The exported record is retained for three months.
|
DataArts Catalog | A maximum of 100 metadata collection tasks can be created in a workspace. Metadata collection tasks can be obtained through DDL SQL statements of the engine. You are not advised to collect more than 1,000 tables through a single task. If necessary, you can create multiple collection tasks. In addition, you need to set the scheduling time and frequency properly based on your requirements to avoid heavy access and connection pressure on the engine. The recommended settings are as follows: If your service requires a metadata validity period of one day, set the scheduling period to max(one day, one-off collection period). This rule also applies to other scenarios. If your service mainly runs in the daytime, set a scheduling time in the night during which the metadata collection has the minimum impact on the data source. This rule also applies to other scenarios.
Only the jobs that are scheduled and executed in DataArts Factory generate data lineages. Tested jobs do not generate data lineages. Historical data connections of the last seven days, 15 days, or 30 days can be displayed on the Dashboard page on the Metadata Collection page.
|