License preemption example

Configuration

This example uses LicenseA as name of preemption resource.

lsf.shared

Add the resource to the Resource section.
Begin Resource
RESOURCENAME TYPE    INTERVAL INCREASING DESCRIPTION
LicenseA     Numeric 60       N          (custom application)
...
End Resource

lsf.cluster.cluster_name

Add the resource to the ResourceMap section
Begin ResourceMap
RESOURCENAME LOCATION
LicenseA     [all]
...
End ResourceMap

lsb.params

Add the resource to the list of preemption resources.
...
PREEMPTABLE_RESOURCES = LicenseA
...

lsb.queues

Define a higher priority queue to be a PREEMPTIVE queue by adding one line in the queue definition.
Begin Queue
QUEUE_NAME=high
PRIORITY=40
...
PREEMPTION=PREEMPTIVE
DESCRIPTION=jobs may preempt jobs in lower-priority queues
...
End Queue
Configure a job control action in a lower priority queue, let SIGTSTP be sent when the SUSPEND action is called, we assume your application can catch the signal SIGTSTP and release the resource (license) it used, then suspend itself. You should also make sure that your application can catch the signal SIGCONT while it is suspended, and consume (check out) the resource (license) again.
Begin Queue
QUEUE_NAME=low
PRIORITY=20
...
JOB_CONTROLS=SUSPEND[SIGTSTP] RESUME[SIGCONT] TERMINATE[SIGTERM]
DESCRIPTION=jobs preempted by jobs in higher-priority queues
...
End Queue

ELIM

Write an ELIM to report the current available number of Application A licenses. This ELIM starts on the master host.

Operation

Check how many LicenseA resources are available

Check the number of LicenseA existing in the cluster by using bhosts -s LicenseA. In this example, 2 licenses are available.
bhosts -s LicenseA
RESOURCE     TOTAL RESERVED LOCATION
LicenseA     2     0.0      hostA hostB ...

Use up all LicenseA resources

Submit 2 jobs to a low-priority queue to consume those 2 licenses.
bsub -J first -q low -R "rusage[LicenseA=1:duration=2]" your_app
bsub -J second -q low -R "rusage[LicenseA=1:duration=2]" your_app
After a while, those jobs are running and the LicenseA resource is used up.
bjobs
JOBID USER STAT QUEUE FROM_HOS EXEC_HOST JOB_NAME SUBMIT_TIME
201   you  RUN  low   hostx    hostA     /first   Aug 23 15:42
202   you  RUN  low   hostx    hostB     /second  Aug 23 15:43
bhosts -s LicenseA
RESOURCE     TOTAL RESERVED LOCATION
LicenseA     0     2.0      hostA hostB ...

Preempt a job for the LicenseA resource

Submit a job to a high-priority queue to preempt a job from a low-priority queue for the resource LicenseA.
bsub -J third -q high -R "rusage[LicenseA=1:duration=2]" your_app
After a while, the third job is running and the second job is suspended.
bjobs
JOBID USER STAT  QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME
203   you  RUN   high  hostx     hostA     /third   Aug 23 15:48
201   you  RUN   low   hostx     hostA     /first   Aug 23 15:42
202   you  SSUSP low   hostx     hostB     /second  Aug 23 15:43
bhosts -s LicenseA 
RESOURCE     TOTAL RESERVED LOCATION
LicenseA     0     2.0      hostA hostB ...