Project mode preemption

Preemption only occurs when there are no free licenses. During preemption, a project releases a borrowed license to the project that owns the license (and now has demand).

Jobs using licenses that support job suspension release their tokens and automatically resume from where they were suspended. Jobs using licenses that do not support suspension are killed and restarted from the beginning.

Preemption only applies to project mode, and depending on your configuration takes the following into consideration:
  • runtime (a job that has the smallest run time gets preempted first, in general)

  • fairshare settings

  • ownership

  • priority

  • minimal job preemption

Depending on how your projects are set up (whether they are all at the same level or not), your preemption is either flat or hierarchical.

Basic preemption with projects configured

When preemption occurs, License Scheduler calculates token usage for each project. The calculation considers tokens in use, tokens required, and token ownership value.

Based on the token usage, License Scheduler determines the projects that require tokens, and those that have too many.

  • Jobs belonging to projects requiring tokens are scheduled first, ordered by project fairshare settings.

  • Jobs belonging to projects with extra tokens are preempted first, if needed, ordered by project fairshare settings and the length os time each job has been running.

With PRIORITY

If project PRIORITY is configured in the Project section, the sort order of projects is based on priority, where a higher priority project is preempted last.

With PREEMPT_ORDER

If PREEMPT_ORDER is set to BY_OWNERSHIP in the Feature section, the projects are sorted by ownership.
  • Projects with the highest ownership are scheduled first.

  • Projects with the smallest ownership are preempted first.

This setting overrides basic preemption and PRIORITY.

With ENABLE_MINJOB_PREEMPTION

If ENABLE_MINJOB_PREEMPTION=Y, the number of preempted jobs is minimized. Projects with extra tokens are sorted by PRIORITY (if configured) or fairshare. The jobs are then sorted by RUSAGE.

Jobs with higher RUSAGE are preempted first to minimize the number of jobs preempted.

This setting is used in addition to basic preemption or PRIORITY.

Hierarchical preemption with project groups configured

When project groups are configured, introducing a hierarchy into the project configuration, hierachical preemption applies.

There are two methods of hierarchical preemption:
  1. Top-down (default): Preemption occurs between cousins rather than siblings. The result is to balance preemption between the entire hierarchy of projects.

  2. Bottom-up (if LS_PREEMPT_PEER=Y): Siblings can preempt each other. The result is to balance preemption within a family of projects first.

For example, your projects are set up as follows:

In top-down preemption, if P8 needs a token, it preempts from P1, P2, or P3 (who are more distant relations), not from P6 or P7 (siblings of P8).

In bottom-up preemption, P8 preempts instead from its siblings (P6 or P7).

Limits

Hierarchical preemption is also affected by any limits placed on the projects. If a limit has already been reached (at any level of the hierarchy), License Scheduler considers the next possible node for preemption instead.