Preempting a Task¶
Scenario¶
The capacity scheduler of ResourceManager implements job preemption to simplify job running in queues and improve resource utilization. The process is as follows:
Assume that there are two queues (Queue A and Queue B). The capacity of Queue A is 25%, and the capacity of Queue B is 75%.
In the initial state, Task 1 is distributed to Queue A for processing, requiring 75% cluster resources. Task 2 is distributed to Queue B for processing, requiring 50% cluster resources.
Task 1 uses 25% cluster resources provided by Queue A and 50% resources from Queue B. Queue B reserves 25% cluster resources.
If task preemption is enabled, the resources of Task 1 will be preempted. Queue B preempts 25% cluster resources from Queue A for Task 2.
Task 1 will be executed when Task 2 is complete and the cluster has sufficient resources.
Procedure¶
Navigation path for setting parameters:
Go to the All Configurations page of Yarn and enter a parameter name in the search box by referring to Modifying Cluster Service Configuration Parameters.
Parameter | Description | Default Value |
|---|---|---|
yarn.resourcemanager.scheduler.monitor.enable | Whether to start scheduler monitoring according to yarn.resourcemanager.scheduler.monitor.policies. If this parameter is set to true, scheduler monitoring is enabled based on policies specified by yarn.resourcemanager.scheduler.monitor.policies and task resource preemption is enabled based on the scheduler information. If this parameter is set to false, scheduler monitoring is disabled. | false |
yarn.resourcemanager.scheduler.monitor.policies | List of the SchedulingEditPolicy class to be used with the scheduler | org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy |
yarn.resourcemanager.monitor.capacity.preemption.observe_only |
| false |
yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval | Monitoring interval, in millisecond. If this parameter is set to a larger value, capacity detection will not be performed frequently. | 3000 |
yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill | Interval between the time when a resource preemption request is sent and the time when the container is stopped (resources are released), in millisecond. The value must be greater than or equal to 0. By default, if ApplicationMaster does not stop the container within 15 seconds, ResourceManager will forcibly stop the container after 15 seconds. | 15000 |
yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round | Maximum resource preemption ratio in a period. This value can be used to limit the speed at which containers are reclaimed from the cluster. After the expected total preemption value is calculated, the policy scales the preemption ratio back to this limit. | 0.1 |
yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity | Resource preemption dead zone = Total number of resources in the cluster x Value of this configuration item + Original resources of a queue (for example, Queue A). When resources actually used by a task in Queue A exceeds the preemption dead zone, the resource beyond the preemption dead zone is preempted. The value range is 0 to 1. Note A smaller value is recommended for effective preemption. | 0 |
yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor | Preemption percentage. Containers preempt only this percentage of the resources. For example, a termination factor of 0.5 will reclaim almost 95% of resources within 5 times of yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill, even in the absence of natural termination. That is, 5 consecutive preemptions will be performed and each time half of the target resources will be preempted. The trend is geometric convergence. The interval of each preemption is yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill. The value range is 0 to 1. | 1 |