HA

Scenarios

The Flink HA mode depends on ZooKeeper. Therefore, ZooKeeper-related configuration items must be set.

Configuration Description

Configuration items include the ZooKeeper address, path, and security certificate.

Table 1 Parameters

Parameter

Description

Default Value

Mandatory

high-availability

Whether HA is enabled. Only the following two modes are supported currently:

  1. none: Only a single JobManager is running. The checkpoint is disabled for JobManager.

  2. ZooKeeper:

    • In non-Yarn mode, multiple JobManagers are supported and the leader JobManager is elected.

    • In Yarn mode, only one JobManager exists.

zookeeper

No

high-availability.zookeeper.quorum

ZooKeeper quorum address.

Automatic configuration

No

high-availability.zookeeper.path.root

Root directory that Flink creates on ZooKeeper, storing metadata required in HA mode.

/flink

No

high-availability.storageDir

Directory for storing JobManager metadata of state backend. ZooKeeper stores only pointers to actual data.

hdfs:///flink/recovery

No

high-availability.zookeeper.client.session-timeout

Session timeout duration on the ZooKeeper client. The unit is millisecond.

60000

No

high-availability.zookeeper.client.connection-timeout

Connection timeout duration on the ZooKeeper client. The unit is millisecond.

15000

No

high-availability.zookeeper.client.retry-wait

Retry waiting time on the ZooKeeper client. The unit is millisecond.

5000

No

high-availability.zookeeper.client.max-retry-attempts

Maximum retry times on the ZooKeeper client.

3

No

high-availability.job.delay

Delay of job restart when JobManager recovers.

The default value is the same as the value of akka.ask.timeout.

No

high-availability.zookeeper.client.acl

ACL (open creator) of the ZooKeeper node. For ACL options, see https://zookeeper.apache.org/doc/r3.5.1-alpha/zookeeperProgrammers.html#sc_BuiltinACLSchemes.

This parameter is configured automatically according to the cluster installation mode.

  • Security mode: The default value is creator.

  • Non-security mode: The default value is open.

Yes

zookeeper.sasl.disable

Simple authentication and security layer (SASL)-based certificate enable switch.

This parameter is configured automatically according to the cluster installation mode.

  • Security mode: The default value is false.

  • Non-security mode: The default value is true.

Yes

zookeeper.sasl.service-name

  • If the ZooKeeper server configures a service whose name is different from ZooKeeper, this configuration item can be set.

  • If service names on the client and server are inconsistent, authentication fails.

zookeeper

Yes

Note

For versions earlier than MRS 3.x, the high-availability.job.delay parameter is not supported.

last updated: 2024-11-14 12:28 UTC - commit: 9d95be538b931742820cff066f12ce18f037ed96