Data Sources¶
Before using DataArts Studio, select a cloud service or data warehouse as the data lake. The data lake stores raw data and data generated during data governance and serves for data development, data services, and data operations. DataArts Studio integrates a wealth of data engines and can connect to cloud data lakes and database cloud services, such as Data Warehouse Service (DWS), Data Lake Insight (DLI), and MapReduce Service (MRS) Hive. It can also connect to traditional enterprise databases, such as MySQL and PostgreSQL.
Data Sources Supported By DataArts Studio¶
Data sources supported by DataArts Studio are classified into data sources supported by DataArts Migration and those supported by other DataArts Studio components.
The DataArts Migration component integrates data into the data lake and supports more types of data sources.
For details on data sources supported by DataArts Migration, see Data Sources Supported by DataArts Migration. To use these data sources, you must create corresponding data links in DataArts Migration, which can only be used in the DataArts Migration module.
The data sources supported by other DataArts Studio components form the data lake base of DataArts Studio.
Table 1 lists the data sources supported by other components. For details, see Overview. To use these data sources in other components, create data connections on the DataArts Studio Management Center console. These data connections cannot be used in the DataArts Migration module.
Data Source Type | Management Center | DataArts Factory |
|---|---|---|
DWS | Supported | Supported |
DLI | Supported | Supported |
MRS HBase | Supported | Not supported |
MapReduce (MRS) Hive | Supported | Supported |
MRS Kafka | Supported | Supported |
MySQL | Supported | Not supported |
MapReduce (MRS) Spark | Supported | Supported |
RDS for MySQL | Supported | Supported |
RDS for PostgreSQL | Supported | Supported |
Host Connection | Supported | Supported |
MapReduce (MRS) Presto | Supported | Supported |
Overview¶
Data Source Type | Description |
|---|---|
DWS | DWS employs the shared-nothing architecture and massively parallel processing (MPP) engine. It is compatible with ANSI SQL 99, SQL 2003, and the PostgreSQL or Oracle database ecosystem, providing competitive solutions for analyzing petabytes of data in various industries. |
DLI | DLI is a serverless big data compute and analysis service that is fully compatible with Apache Spark and Apache Flink ecosystems. With multi-model engines supported by DLI, enterprises can use SQL statements or programs to easily complete batch processing, stream processing, in-memory computing, and machine learning of heterogeneous data sources. |
MRS HBase | HBase undertakes data storage. It is an open-source, column-oriented, distributed storage system that is suitable for storing massive amounts of unstructured or semi-structured data. It features high reliability, high performance, and flexible scalability, and supports real-time data read/write. MRS HBase stores massive amount of data and supports data queries in milliseconds. MRS HBase can load and update logistics data in milliseconds, and query and analyze petabytes of time series data in seconds. |
MRS Hive | Hive is a mechanism that can store, query, and analyze large-scale data stored in Hadoop. Hive defines simple SQL-like query language, which is known as HiveQL. It allows users familiar with SQL to query data. MRS Hive can be used to analyze terabytes or petabytes of data and quickly migrate on-premises Hadoop big data platforms (such as CDH and HDP) to the cloud without service interruption and service code modification. |
MRS Kafka | MRS provides dedicated MRS Kafka clusters. Kafka is an open-source, distributed, partitioned, and replicated commit log service. Kafka is publish-subscribe messaging, rethought as a distributed commit log. It provides features similar to Java Message Service (JMS) but another design. It features message endurance, high throughput, distributed methods, multi-client support, and real time. It applies to both online and offline message consumption, such as regular message collection, website activeness tracking, aggregation of statistical system operation data (monitoring data), and log collection. These scenarios engage large amounts of data collection for Internet services. |
MySQL | MySQL is one of the most popular open-source databases. It features excellent performance, uses mature and stable architecture, supports popular applications, adapts to multiple fields and industries, and supports various web applications. It is cost-effective and preferred by small- and medium-sized enterprises. |
MRS Spark | Spark is an open-source parallel data processing framework. It helps users easily develop unified big data applications and perform cooperative processing, stream processing, and interactive analysis on data. Spark provides a framework featuring fast calculation, write, and interactive query. Spark has obvious advantages over Hadoop in terms of performance. Spark provides the Spark SQL language similar to SQL statements to process structured data. |
RDS | RDS is an online, out-of-the-box relational database service that is based on the cloud computing platform. It is stable, reliable, scalable, and easy to manage. Currently, DataArts Studio supports only MySQL and PostgreSQL databases in RDS. |
Host Connection | You can connect to a specified host during data development and execute shell or Python scripts on the host through script development and job development. If the host connection information changes, you only need to edit it on the Host Connections page, but do not need to edit it in scripts or jobs one by one. |
MRS Presto | Presto is an open-source SQL query engine for running interactive analytic queries against data sources of all sizes. It applies to massive structured/semi-structured data analysis, massive multi-dimensional data aggregation/report, ETL, ad-hoc queries, and more scenarios. Presto allows querying data where it lives, including HDFS, Hive, HBase, Cassandra, relational databases, or even proprietary data stores. A Presto query can combine different data sources to perform data analysis across the data sources. |