Introduction to Storage-Compute Decoupling

In scenarios that require large storage capacity and elastic compute resources, MRS enables you to store data in OBS and use an MRS cluster for data computing only. In this way, storage and compute are separated.

Note

In the big data decoupled storage-compute scenario, the OBS parallel file system must be used to configure a cluster. Using common object buckets will greatly affect the cluster performance.

Process of using the storage-compute decoupling function:

1. Configure a storage-compute decoupled cluster using either of the following methods (agency is recommended):

   • Bind an agency of the ECS type to an MRS cluster to access OBS, preventing the AK/SK from being exposed in the configuration file. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency).

   • Configure the AK/SK in an MRS cluster. The AK/SK will be exposed in the configuration file in plaintext. Exercise caution when performing this operation. For details, see Configuring a Storage-Compute Decoupled Cluster (AK/SK).

2. Use the cluster.

   For details, see the following sections':

   • Interconnecting Flink with OBS

   • Interconnecting Flume with OBS

   • Interconnecting HDFS with OBS

   • Interconnecting Hive with OBS

   • Interconnecting MapReduce with OBS

   • Interconnecting Spark2x with OBS

   • Interconnecting Sqoop with External Storage Systems

last updated: 2024-05-02 02:56