Creating a Custom Cluster¶

The first step of using MRS is to create a cluster. This section describes how to create a cluster on the Custom Config tab of the MRS management console.

You can create an IAM user or user group on the IAM management console and grant it specific operation permissions, to perform refined resource management after registering an account. For details, see Creating an MRS User.

Log in to the MRS console.
Click Create Cluster. The page for creating a cluster is displayed.
Note
When creating a cluster, pay attention to quota notification. If a resource quota is insufficient, increase the resource quota as prompted and create a cluster.
Click the Custom Config tab.
Configure cluster information by referring to Software Configurations and click Next.
Configure cluster information by referring to Hardware Configurations and click Next.
Set advanced options by referring to Advanced Options and click Next.
If Kerberos authentication is enabled for a cluster, check whether Kerberos authentication is required. If yes, click Continue. If no, click Back to disable Kerberos authentication and then create a cluster.
Note
For any doubt about the pricing, click Pricing details in the lower left corner.
On the Confirm Configuration page, check the cluster configuration information. If you need to adjust the configuration, click to go to the corresponding tab page and configure parameters again.
Select the checkbox to enable secure communications. For details, see Communication Security Authorization.
Click Create Now.
If Kerberos authentication is enabled, check whether this function is required. If it is, click Continue. If not, click Back to disable it and then proceed with the subsequent step. This option cannot be changed after you create a cluster.
Click Back to Cluster List to view the cluster status.
For details about cluster status during creation, see the description of the status parameters in Table 1.
It takes some time to create a cluster. The initial status of the cluster is Starting. After the cluster has been created successfully, the cluster status becomes Running.
On the MRS management console, a maximum of 10 clusters can be concurrently created, and a maximum of 100 clusters can be managed.

Software Configurations¶

**Table 1** MRS cluster software configuration¶
Parameter	Description
Region	Select a region. Cloud service products in different regions cannot communicate with each other over an intranet. For low network latency and quick access, select the nearest region.
Cluster Name	The cluster name must be unique. A cluster name can contain 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed. The default name is mrs_xxxx. xxxx is a random collection of letters and digits.
Cluster Type	The cluster types are as follows: Analysis cluster: is used for offline data analysis and provides Hadoop components. Streaming cluster: is used for streaming tasks and provides stream processing components. Hybrid cluster: is used for both offline data analysis and streaming processing and provides Hadoop components and streaming processing components. You are advised to use a hybrid cluster to perform offline data analysis and streaming processing tasks at the same time. Custom: You can adjust the cluster service deployment mode based on service requirements. For details, see Creating a Custom Cluster. (This type is currently available only in MRS 3.x.) Doris cluster: is a high-performance and real-time analytical database for report analysis, ad hoc queries, and acceleration of data lake federation queries. Note MRS streaming clusters do not support job and file management functions. To install all components in a cluster, select Custom.
Cluster Version	Currently, MRS 3.1.2-LTS.6, MRS 3.2.0-LTS.2 and MRS 3.3.0-LTS.1 are supported. The default value displayed on the UI varies depending on the version.
Cluster Type	The cluster types are as follows: Analysis cluster: is used for offline data analysis and provides Hadoop components. Streaming cluster: is used for streaming tasks and provides stream processing components. Hybrid cluster: is used for both offline data analysis and streaming processing and provides Hadoop components and streaming processing components. You are advised to use a hybrid cluster to perform offline data analysis and streaming processing tasks at the same time. Custom: You can adjust the cluster service deployment mode based on service requirements. For details, see Creating a Custom Topology Cluster. Note MRS streaming clusters do not support job and file management functions. To install all components in a cluster, select Custom.
Components	MRS cluster components. For details about component versions supported by different versions of MRS clusters, see List of MRS Component Versions.
Component Port	Use the default Open source.

Hardware Configurations¶

**Table 2** MRS cluster hardware configuration¶
Parameter	Description
AZ	Select the AZ associated with the region of the cluster. An AZ is a physical area that uses independent power and network resources. AZs are physically isolated but interconnected through the internal network. This improves the availability of applications. You are advised to create clusters in different AZs.
VPC	A VPC is a secure, isolated, and logical network environment. Select the VPC for which you want to create a cluster and click View VPC to view the name and ID of the VPC. If no VPC is available, create one.
Subnet	A subnet provides dedicated network resources that are isolated from other networks, improving network security. Select the subnet for which you want to create a cluster. Click View Subnet to view details about the selected subnet. If no subnet is created in the VPC, go to the VPC console and choose Subnets > Create Subnet to create one. For details about how to configure network ACL outbound rules, see How Do I Configure a Network ACL Outbound Rule? Note In MRS, IP addresses are automatically assigned to clusters during cluster creation basically based on the following formula: Quantity of IP addresses = Number of cluster nodes + 2 (one for Manager; one for the DB). In addition, if the Hadoop, Hue, Sqoop, and Presto or Loader and Presto components are selected during cluster deployment, one IP address is added for each component. To create a ClickHouse cluster independently, the number of IP addresses required is calculated as follows: Number of IP addresses = Number of cluster nodes + 1 (for Manager).
Security Group	A security group is a set of ECS access rules. It provides access policies for ECSs that have the same security protection requirements and are mutually trusted in a VPC. When you create a cluster, you can select Auto create from the drop-down list of Security Group to create a security group or select an existing security group. Note When you select a security group created by yourself, ensure that the inbound rule contains a rule in which Protocol is set to All, Port is set to All, and Source is set to a trusted accessible IP address range. Do not use 0.0.0.0/0 as a source address. Otherwise, security risks may occur. If you do not know the trusted accessible IP address range, select Auto create.
EIP	After binding an EIP to an MRS cluster, you can use the EIP to access the Manager web UI of the cluster. When creating a cluster, you can select an available EIP from the drop-down list and bind it. If no EIP is available in the drop-down list, click Manage EIP to access the EIPs service page to create one. Note This parameter is valid only in MRS 1.8.0 or later. The EIP must be in the same region as the cluster.

**Table 3** Cluster node information¶
Parameter	Description
CPU Architecture	CPU architecture supported by MRS. x86: The x86-based CPU architecture uses Complex Instruction Set Computing (CISC). Each instruction can be used to execute low-level hardware operations. The number of instructions is large, and the length of each instruction is different. Therefore, executing such an instruction is complex and time-consuming.
Common Node Configurations	This parameter is valid only when Cluster Type is set to Custom. For details, see Custom Cluster Template Description.
Node Type	MRS provides three types of nodes: Master: A Master node in an MRS cluster manages the cluster, assigns executable cluster files to Core nodes, traces the execution status of each job, and monitors the DataNode running status. Core: A Core node in a cluster processes data and stores process data in HDFS. Analysis Core nodes are created in an analysis cluster. Streaming Core nodes are created in a streaming cluster. Both analysis and streaming Core nodes are created in a hybrid cluster. Task: A Task node in a cluster is used for computing and does not store persistent data. Yarn and Storm are mainly installed on Task nodes. Task nodes are optional, and the number of Task nodes can be zero. Analysis Task nodes are created in an analysis cluster. Streaming Task nodes are created in a streaming cluster. Both analysis and streaming Task nodes are created in a hybrid cluster. When the data volume change is small in a cluster but the cluster's service processing capabilities need to be remarkably and temporarily improved, add Task nodes to address the following situations: Service volumes temporarily increase, for example, report processing at the end of the year. Long-term tasks must be completed in a short time, for example, some urgent analysis tasks.
Instance Specifications	Instance specifications of Master or Core nodes. MRS supports host specifications determined by CPU, memory, and disk space. Click to configure the instance specifications, system disk, and data disk parameters of the cluster node. Note More advanced instance specifications provide better data processing. If you select non-HDD disks for Core nodes, the disk types of Master and Core nodes are determined by Data Disk. The memory of the master node must be greater than 64 GB.
System Disk	Storage type and storage space of the system disk on a node. Storage type can be any of the following: SATA: common I/O SAS: high I/O SSD: ultra-high I/O GPSSD: general-purpose SSD
Data Disk	Data disk storage space of a node. To increase data storage capacity, you can add disks at the same time when creating a cluster. The following two application scenarios are involved. Data storage and computing are separated. Data is stored in OBS, which features low cost and unlimited storage capacity. The clusters can be terminated at any time in OBS. The computing performance is determined by OBS access performance and is lower than that of HDFS. This configuration is recommended if data computing is infrequent. Data storage and computing are not separated. Data is stored in HDFS, which features high cost, high computing performance, and limited storage capacity. Before terminating clusters, you must export and store the data. This configuration is recommended if data computing is frequent. The storage type can be any of the following: SATA: common I/O SAS: high I/O SSD: ultra-high I/O GPSSD: general-purpose SSD Note More nodes in a cluster require higher disk capacity of Master nodes. To ensure stable cluster running, set the disk capacity of the Master node to over 600 GB if the number of nodes is 300 and increase it to over 1 TB if the number of nodes reaches 500.
Topology Adjustment	If the deployment mode in the Common Node does not meet the requirements, set Topology Adjustment to Enable and adjust the instance deployment mode based on service requirements. For details, see Topology Adjustment for a Custom Cluster. This parameter is valid only when Cluster Type is set to Custom.

Advanced Options¶

**Table 4** MRS cluster advanced configuration topology¶
Parameter	Description
Kerberos Authentication	Whether to enable Kerberos authentication when logging in to Manager. This option cannot be changed after you create a cluster. : If Kerberos Authentication is disabled, common users can use all functions of an MRS cluster. You are advised to disable Kerberos authentication in single-user scenarios. : If Kerberos Authentication is enabled, common users cannot use the file and job management functions of an MRS cluster and cannot view cluster resource usage or the job records for Hadoop and Spark. To use more cluster functions, the users must contact the Manager administrator to assign more permissions. You are advised to enable Kerberos authentication in multi-user scenarios. Currently, Presto does not support Kerberos authentication.
Username	Name of the administrator of Manager. admin is used by default.
Password	Password of the Manager administrator The following requirements must be met: Must contain 8 to 26 characters. Must contain at least four of the following: Lowercase letters Uppercase letters Digits At least one of the following special characters: `~!@#$%^&*()-_=+\|[{}];:',<.>/? Cannot be the same as the username or the username spelled backwards. Password Strength: The colorbar in red, orange, and green indicates weak, medium, and strong password, respectively.
Confirm Password	Enter the password of the Manager administrator again.
Login Mode	Password Log in to the ECS or BMS as user root. Enter the password of user root and confirm the password. A password must meet the following requirements: Must be a string and 8 to 26 characters long. Must contain at least four of the following: uppercase letters, lowercase letters, digits, and special characters (`~!@#$%^&()-_=+\|[{}];:',<.>/?). The password cannot be the username or the reverse username. Key Pair Key pairs are used to log in to ECS nodes of the cluster. Select a key pair from the drop-down list. Select "I acknowledge that I have obtained private key file SSHkey-xxx* and that without this file I will not be able to log in to my ECS." If you have never created a key pair, click View Key Pair to create or import a key pair. And then, obtain a private key file. A key pair, also called an SSH key, consists of a public key and a private key. You can create an SSH key and download the private key for authenticating remote login. For security, a private key can only be downloaded once. Keep it secure. Use an SSH key in either of the following two methods: Creating an SSH key: After you create an SSH key, a public key and a private key are generated. The public key is stored in the system, and the private key is stored in the local ECS. When you log in to an ECS, the public and private keys are used for authentication. Importing an SSH key: If you have obtained the public and private keys, import the public key into the system. When you log in to an ECS, the public and private keys are used for authentication.
Hostname Prefix	Enter the prefix for the computer hostname of an ECS or BMS in the cluster.
Setting Advanced Options	Advanced function parameters of an MRS cluster. Select Configure. For details, see Table 5.

**Table 5** (Optional) Advanced configuration information of the MRS cluster¶
Parameter	Description
Tag	For details, see Adding a Tag to a Cluster/Node.
Auto Scaling	Auto scaling can be configured only after you specify task node specifications in the Configure Hardware step by referring to Configuring an Auto Scaling Rule.
Bootstrap Action	For details, see Adding a Bootstrap Action.
Agency	By binding an agency, ECSs or BMSs can manage some of your resources. Determine whether to configure an agency based on the actual service scenario. For example, you can configure an agency of the ECS type to automatically obtain the AK/SK to access OBS. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency). The MRS_ECS_DEFAULT_AGENCY agency has the OBSOperateAccess permission of OBS and the CESFullAccess (for users who have enabled fine-grained policies), CES Administrator, and KMS Administrator permissions in the region where the cluster is located.
Data Disk Encryption	Whether to encrypt data in the data disk mounted to the cluster. This function is disabled by default. To use this function, you must have the Security Administrator and KMS Administrator permissions. This parameter is not available for MRS 3.1.0 and MRS 3.1.2-LTS.3. Keys used by encrypted data disks are provided by the Key Management Service (KMS) of the Data Encryption Workshop (DEW), secure and convenient. Therefore, you do not need to establish and maintain the key management infrastructure. Click Data Disk Encryption to enable or disable the data disk encryption function.
Data Disk Key ID	This parameter is displayed only when the Data Disk Encryption function is enabled. This parameter indicates the key ID corresponding to the selected key name. This parameter is not available for MRS 3.1.0 and MRS 3.1.2-LTS.3.
Data Disk Key Name	This parameter is mandatory when the Data Disk Encryption function is enabled. Select the name of the key used to encrypt the data disk. By default, the default master key named evs/default is selected. You can select another master key from the drop-down list. This parameter is not available for MRS 3.1.0 and MRS 3.1.2-LTS.3. If disks are encrypted using a CMK, which is then disabled or scheduled for deletion, the disks can no longer be read from or written to, and data on these disks may never be restored. Exercise caution when performing this operation. Click View Key List to enter a page where you can create and manage keys.
Alarm	If the alarm function is enabled, the cluster maintenance personnel can be notified in a timely manner to locate faults when the cluster runs abnormally or the system is faulty.
Rule Name	Name of the rule for sending alarm messages. The value can contain only digits, letters, hyphens (-), and underscores (_).
Topic Name	Select an existing topic or click Create Topic to create a topic. To deliver messages published to a topic, you need to add a subscriber to the topic. For details, see Adding Subscriptions to a Topic. A topic serves as a message sending channel, where publishers and subscribers can interact with each other.
Logging	Whether to collect logs when cluster creation fails. After the logging function is enabled, system logs and component run logs are automatically collected and saved to the OBS file system in scenarios such as cluster creation failures and scale-out or scale-in failures for O&M personnel to quickly locate faults. The log information is retained for a maximum of seven days.

Failed to Create a Cluster¶

If a cluster fails to be created, the failed task will be managed on the Manage Failed Tasks page. Choose Clusters > Active Clusters. Click shown in Figure 1 to go to the Manage Failed Tasks page. In the Status column, hover the cursor over to view the failure cause. You can delete failed tasks by referring to Viewing Failed MRS Tasks.

Table 6 lists the error codes of MRS cluster creation failures.

**Table 6** Error codes¶
Error Code	Description
MRS.101	Insufficient quota to meet your request. Contact customer service to increase the quota.
MRS.102	The token cannot be null or invalid. Try again later or contact customer service.
MRS.103	Invalid request. Try again later or contact customer service.
MRS.104	Insufficient resources. Try again later or contact customer service.
MRS.105	Insufficient IP addresses in the existing subnet. Try again later or contact customer service.
MRS.201	Failed due to an ECS error. Try again later or contact customer service.
MRS.202	Failed due to an IAM error. Try again later or contact customer service.
MRS.203	Failed due to a VPC error. Try again later or contact customer service.
MRS.400	MRS system error. Try again later or contact customer service.

last updated: 2024-11-14 12:28 UTC - commit: 9d95be538b931742820cff066f12ce18f037ed96