Resource Reclaim and Distribution Behavior

Reclaim behavior(default and feature enabled)


Default behavior (feature not enabled)

Feature enabled

When to reclaim

  • If a consumer is not using up to its share and now has new demand, the consumer reclaims slots from other consumers who borrowed from the share pool.

  • Same behavior as with feature not enabled.

From which consumer to reclaim

  • Configured share ratio is a minimum guarantee.

  • Configured share ratio is a mininum guarantee.

  • Share quota (the number of slots a consumer can possibly get) is always fixed, equal to the configured share for the consumer.

  • Share quota (the number of slots a consumer can possibly get) is dynamic, it is the configured share plus any slots not used by other consumers. When other consumers do not have demand, the share quota of consumers with demand increases.

  • Slots are reclaimed from any consumers who borrowed. If a consumer is not using more than its share ratio, no slots can be reclaimed from the consumer.

  • Slots are reclaimed only from consumers who borrowed more than their share.

  • A consumer cannot reclaim slots from other consumers who borrow. *****

  • From which consumer to reclaim first depends on theconsumer tree --- if the parent is using more than its share, slots fromthe child consumer are more likely to be reclaimed than from anotherconsumer whose parent is not using more than its share.

  • From which consumer to reclaim first depends on the share ratio across the entrire consumer tree. The consumer that has borrowed the most when evaluated against its share ratio is reclaimed first.

    To identify the consumer who borrowed the most against its share ratio, the system evaluates from top to bottom of the consumer tree. Parents are evaluated first.

    For example, when comparing two parents and parent A has borrrowed more slots than parent B according to its share ratio, the system then evaluates the children of parent A to identify the child which has borrowed the most according to its share ratio. The system will then compare the children of parent A against each other to see from which one to reclaim slots.

If two parent consumers are using more than their share, resources are reclaimed from the lowest priority parent first.


Distribution behavior(default and feature enabled)


Default behavior (feature not enabled)

Feature enabled

  • When a consumer has demand, the consumer is allocated slots up to its share.

    The number of slots a consumer can borrow is limited by its share.

  • When a consumer has demand, the consumer is allocated slots up to its share. If it has more demand, the consumer can additionally borrow slots from the share pool that are not used by other consumers.

The number of slots a consumer can borrow depends on cluster workload. Consumers without demand are not included in calculation of share quota.

Slots are distributed among consumers with demand according to their share ratio.

  • When a consumer does not have demand, its share goes to the share pool. Slots in the share pool can be borrowed by any consumer, in First-Come, First-Servedorder.

  • When a consumer does not have demand, its share goes to the share pool. Slots in the share pool can be borrowed by consumers with demand according to their share ratio.

  • If two consumers request resources at the same time, the highest priority consumer will get all slots in the share pool if it has enough demand.

    If two consumers request resources at the same time and the highest priority consumer does not have enough demand to get all slots in the share pool, the next consumer with the higher priority will get whichever number of slots is left over.

  • If two consumers request resources at the same time, consumers are allocated slots according to their share ratio.

  • Whenever a consumer borrows from the share pool, theborrowed resources can be reclaimed if other consumers have demand only if other consumers are not using their share.

  • Whenever a consumer borrows from the share pool, the borrowed resources can be reclaimed according to share ratio if other consumers have demand, even if the consumer is not using its share.


Examples: How resource reclaim works when not based on share ratio (feature not enabled)

The default behavior for resource reclaim is to reclaim resources according to consumer priority. Resources are reclaimed from the lowest priority consumer. Share ratio is not taken into account for resource reclaim.

Scenario: Three Consumers, 12 slots

Total share pool: 12 slots

There are three consumers: A, B, C. There are no subconsumers.

The lowest priority consumer is consumer B.

The share ratio is equal for all consumers (1:1:1).

Example: Demand exists for all consumers

All consumers have demand for 4 slots, so the total share pool is distributed among all consumers. Each consumer is allocated 4 slots.



Example: No demand for consumer C

Building on the previous example, all consumers are now allocated 4 slots.

Consumer A now has demand for 7 slots.

Consumer B now has demand for 5 slots.

Consumer C no longer has demand.

Because Consumer C does not have demand, its 4 slots can now be allocated among the other consumers. Consumer A is allocated 3 additional slots, consumer B, 1 additional slot.



Example: Demand exists again for Consumer C

Building on the previous example:

Consumer A has demand for 7 slots, allocated 7 slots.

Consumer B has demand for 5 slots, allocated 5 slots.

Consumer C now has demand for 1 slot.

To determine from which consumer slots should be reclaimed, the system looks at the lowest priority consumer that is allocated more than its share (Consumer B), and reclaims 1 slot to allocate to Consumer C.