Using license feature locality

License feature locality allows you to limit features from different service domains to a specific cluster, so that License Scheduler does not grant tokens to jobs from license that legally cannot be used on the cluster requesting the token.

Example configuration: 2 sites and 4 service domains

Some of your service domains may have geographical restrictions when serving licenses. In this example, two clusters in one location can run hspice jobs. and 4 service domains are defined for the hpsice feature:

  • SD1 is a local license file for clusterA with 25 hspice licenses

  • SD2 is a local license file for clusterB with 65 hspice licenses

  • SD3 is a WANable license with 15 hspice licenses

  • SD4 is a globally WANable license with 7 hspice licenses

The geographical license checkout restrictions are:

  • Jobs in clusterA can check out licenses from SD1 SD3 and SD4 but not SD2

  • Jobs in clusterB can check out licenses from SD2 SD3 and SD4 but not SD1

Configuring license feature locality (LOCAL_TO)

Use LOCAL_TO in the feature configuration in lsf.licensescheduler to configure token locality for the license feature. You must configure different feature sections for same feature based on their locality.

LOCAL_TO allows you to limit features from different service domains to specific clusters, so License Scheduler only grants tokens of a feature to jobs from clusters that are entitled to them.

By default, if LOCAL_TO is not defined, the feature is available to all clients and is not restricted by geographical location. When LOCAL_TO is configured, for a feature, License Scheduler treats license features served to different locations as different token names, and distributes the tokens to projects according the distribution and allocation policies for the feature.

For example, if your license servers restrict the serving of license tokens to specific geographical locations, use LOCAL_TO to specify the locality of a license token if any feature cannot be shared across all the locations. This avoids having to define different distribution and allocation policies for different service domains, and allows hierarchical group configurations.

License Scheduler manages features with different localities are different resources. Use blinfo, blusers, and blstat to see the different resource information for the features depending on their cluster locality.

License features with different localities must be defined in different feature sections. The same Service Domain can appear only once in the configuration for a given license feature.

A configuration like LOCAL_TO=Site1(clusterA clusterB) configures the feature for more than one cluster.

A configuration like LOCAL_TO=clusterA configures locality for only one cluster. This is the same as LOCAL_TO=clusterA(clusterA).

Cluster names must be the names of clusters defined in the Clusters section of lsf.licensescheduler.

Examples:

Begin Feature
NAME = hspice
DISTRIBUTION = SD1 (Lp1 1 Lp2 1)
LOCAL_TO = siteUS(clusterA clusterB)
End Feature
Begin Feature
NAME = hspice
DISTRIBUTION = SD2 (Lp1 1 Lp2 1)
LOCAL_TO = clusterA
End Feature
Begin Feature
NAME = hspice
DISTRIBUTION = SD3 (Lp1 1 Lp2 1) SD4 (Lp1 1 Lp2 1)
End Feature

Or use the hierarchical group configuration (GROUP_DISTRIBUTION):

Begin Feature
NAME = hspice
GROUP_DISTRIBUTION = group1
SERVICE_DOMAINS = SD1
LOCAL_TO = siteUS(clusterA clusterB)
End Feature
Begin Feature
NAME = hspice
GROUP_DISTRIBUTION = group1
SERVICE_DOMAINS = SD2
LOCAL_TO = clusterA
End Feature
Begin Feature
NAME = hspice
GROUP_DISTRIBUTION = group1
SERVICE_DOMAINS = SD3 SD4
End Feature

How locality works

When LOCAL_TO is specified in the feature definition in lsf.licensescheduler, license resources requested from different clusters are mapped to different tokens in License Scheduler

You must make sure that your features are configured so that the applications always first tries to checkout licenses locally.

Features with different locality are treated a different tokens by License Scheduler. You must configure separate feature sections for same feature with different localities. For example, feature hspice, because of locality, comprises three different tokens, hspice@clusterA, hspice@clusterB, and hspice (without locality).

How job license demand is passed to License Scheduler

When License Scheduler receives license requests from LSF, it knows where the request is from, and it will interpret the request into demands for tokens usable by that cluster. For example, if clusterA sends a request to bld asking for 1 hspice license, License Scheduler marks the demand for both hspice@clusterA and hspice. When the job gets either token to run, the demand will be cleaned up for both tokens.

Submitting jobs to use license feature locality

When LOCAL_TO is specified for a feature, job submission is simplified. To request a particular license, specify the resource usage string with the same resource name you see in bhosts -s. No OR rusage string is needed. For example:

bsub -Lp Lp1 -R "rusage[hspice=1]" myjob

Viewing feature locality information

When LOCAL_TO is configured for a feature in lsf.licensescheduler, blinfo shows general cluster locality information and distribution for the features.

blinfo
FEATURE          SERVICE_DOMAIN  TOTAL  DISTRIBUTION
hspice           SD3             15     [Lp1, 50.0%] [Lp2, 50.0%]
hspice           SD4             7      [Lp1, 50.0%] [Lp2, 50.0%]
hspice@clusterA  SD1             25     [Lp1, 50.0%] [Lp2, 50.0%]
hspice@siteB     SD2             65     [Lp1, 50.0%] [Lp2, 50.0%]

When LOCAL_TO is configured for a feature in lsf.licensescheduler, blinfo -A shows the feature allocation by cluster locality.

blinfo -A
FEATURE          PROJECT   ALLOCATION
hspice           Lp1       [clusterA, 25.0%] [clusterB, 25.0%]
                           [clusterC, 25.0%] [interactive, 25.0%])
                 Lp2       [clusterA, 50.0%] [clusterB, 50.0%])
hspice@clusterA  Lp1       [clusterA, 100.0%])
                 Lp2       [clusterA, 100.0%])
hspice@siteB     Lp1       [clusterB, 80.0%] [clusterC, 20%])
                 Lp2       [clusterB, 80.0%] [clusterC, 20%])
hspice@clusterC  Lp1       [clusterC, 60.0%] [interactive, 40.0%)
                 Lp2       [clusterC, 60.0%] [interactive, 40.0%)
                 Lp3       [clusterC, 60.0%] [interactive, 40.0%)
vcs              Lp1       [clusterA, 33.0%] [clusterB, 33.0%]
                           [interactive, 33.0%])
                 Lp2       [clusterA, 50.0%] [clusterB, 50.0%])
vcs@clusterA     Lp1       [clusterA, 100.0%])
                 Lp2       [clusterA, 100.0%])
vcs@siteB        Lp1       [clusterB, 80.0%] [clusterC, 20%])
                 Lp2       [clusterB, 80.0%] [clusterC, 20%])
vcs@clusterC     Lp1       [clusterC, 60.0%] [interactive, 40.0%)
                 Lp2       [clusterC, 60.0%] [interactive, 40.0%)
                 Lp3       [clusterC, 60.0%] [interactive, 40.0%)

When LOCAL_TO is configured for a feature, blinfo -C shows the cluster locality information for the features.

blinfo -C
NAME: hspice     FLEX_NAME: hspice
 CLUSTER_NAME    FEATURE          SERVICE_DOMAINS
 clusterA        hspice           SD3 SD4
                 hspice@clusterA  SD1
 clusterB        hspice           SD3 SD4
                 hspice@siteB     SD3
 clusterC        hspice           SD3 SD4
                 hspice@siteB     SD3
                 hspice@clusterC  SD5
NAME: vcs        FLEX_NAME: VCS_Runtime
 CLUSTER_NAME    FEATURE          SERVICE_DOMAINS
 clusterA        vcs              SD3 SD4
                 vcs@clusterA     SD1
 clusterB        vcs              SD3 SD4
                 vcs@siteB        SD3
 clusterC        vcs              SD3 SD4
                 vcs@siteB        SD3
                 vcs@clusterC     SD5

blusers

When LOCAL_TO is configured for a feature in lsf.licensescheduler, blusers shows the cluster locality information for the features.

blusers
FEATURE          SERVICE_DOMAIN  USER     HOST       NLICS    NTASKS
hspice@clusterA  SD1             user1    host1      1        1
hspice@siteB     SD2             user2    host2      1        1

blstat

When LOCAL_TO is configured for a feature in lsf.licensescheduler, blstat shows the cluster locality information for the features.

With the group distribution configuration:

blstat
FEATURE: hspice
 SERVICE_DOMAIN: SD3 SD4
 TOTAL_INUSE: 0    TOTAL_RESERVE: 0    TOTAL_FREE: 22   OTHERS: 0

PROJECT SHARE OWN INUSE RESERVE FREE DEMAND

Lp1 50.0 % 0 0 0 11 0

Lp2 50.0 % 0 0 0 11 0

FEATURE: hspice@clusterA
 SERVICE_DOMAIN: SD1
 TOTAL_INUSE: 0    TOTAL_RESERVE: 0    TOTAL_FREE: 25   OTHERS: 0

PROJECT SHARE OWN INUSE RESERVE FREE DEMAND

Lp1 50.0 % 0 0 0 12 0

Lp2 50.0 % 0 0 0 13 0

FEATURE: hspice@siteB
 SERVICE_DOMAIN: SD2
 TOTAL_INUSE: 0    TOTAL_RESERVE: 0    TOTAL_FREE: 65   OTHERS: 0

PROJECT SHARE OWN INUSE RESERVE FREE DEMAND

Lp1 50.0 % 0 0 0 32 0

Lp2 50.0 % 0 0 0 33 0

bhosts -s

When LOCAL_TO is configured for a feature in lsf.licensescheduler, bhosts -s shows different resource information depending on the cluster locality of the features.

From clusterA:

bhosts -s
RESOURCE                 TOTAL       RESERVED       LOCATION
hspice                   36.0        0.0            host1

From clusterB in siteB:

bhosts -s
RESOURCE                 TOTAL       RESERVED       LOCATION
hspice                   76.0        0.0            host2

How LOCAL_TO works with the other feature section parameters

The following table shows various combinations of LOCAL_TO and other feature section parameters:


NAME

FLEX_NAME

1

AppX

-

2

AppZ201

201-AppZ

3

AppB_v1

AppB


  1. The feature name is same as the real FLEXnet name. Without LOCAL_TO, the feature name AppX cannot be duplicated in feature section. Only one feature section can contain the NAME=AppX.

  2. LSF does not support names that start with a number, or names containing a dash or hyphen character (-), even though they are valid FLEXnet feature names. For these feature names, you must set both NAME and FLEX_NAME in the Features section of lsf.licensescheduler. FLEX_NAME is the actual FLEXnet feature name, and NAME is an arbitrary license token name you choose. Without LOCAL_TO, NAME and FLEX_NAME cannot be in more than one feature section.

  3. You can define different License Scheduler tokens for the same FLEXnet feature. The the service domain names (in either the DISTRIBUTIONS line or the SERVICE_DOMAINS for group configurations) of the same FLEXnet feature in different feature sections must be exclusive. They cannot overlap.

  4. When LOCAL_TO is configured for a feature, you can define different License Scheduler tokens for the same FLEXnet feature with different localities. The constraints are:

    • For the same FLEXnet feature, service domains must be exclusive.

    • The location name of LOCAL_TO defines the locality of that feature, so the name must be unique for all tokens with same FLEXnet feature.

    • You should use same location name for different FLEXnet features with the same pattern of locality, but License Scheduler will not check whether the same location name of a different feature contains the same list of clusters.

  1. Features must either have a different NAME or have LOCAL_TO defined. The service domains for each LS token of same FLEXnet feature must be exclusive.

How LOCAL_TO works with ALLOCATION and ENABLE_INTERACTIVE

The LOCAL_TO parameter simplifies the ALLOCATION configuration. Most of the time you are only interested in who can participate to share a particular token. LOCAL_TO gives the equal share for all the clusters defined in LOCAL_TO and applies to all the projects. Use ALLOCATION to fine tune the shares for individual projects between different clusters:

  • Except for the keyword interactive, all the cluster names defined in ALLOCATION must also be defined in the LOCAL_TO parameter.

  • The global parameter ENABLE_INTERACTIVE and ALLOCATION with interactive share defined works same as before. If ALLOCATION is configured, it ignores the global setting of the ENABLE_INTERACTIVE parameter.

  • If ALLOCATION is not defined, but LOCAL_TO is defined, the default value for ALLOCATION will be equal shares for all the clusters defined in LOCAL_TO parameter. This applies to all license projects defined in DISTRIBUTION or GROUP_DISTRIBUTION.

  • If both ALLOCATION and LOCAL_TO are defined, ALLOCATION parameter can be used to fine tune the shares between the clusters for different projects.

The following table shows example configurations with two clusters and 12 hspice licenses distributed as follows:

DISTRIBUTION = LanServer (Lp1 1 Lp2 1) 

ENABLE_INTERACTIVE

LOCAL_TO

ALLOCATION

No

SiteA(clusterA interactive)

No

clusterA

Lp1(clusterA 0 clusterB 1)

No

clusterA

Lp1(clusterA 1)\

Lp2(clusterA 1)


About interactive taskman jobs

The License Scheduler command taskman is a job starter for interactive jobs to use License Scheduler without bsub. taskman checks out a license token and manages interactive UNIX applications. If LOCAL_TO is specified for a feature, taskman jobs need to specify feature names with locality information similar to submission with bsub. You need to know which token can be used from the location where task is going to run. For example:

taskman -Lp P1 -R "rusage[hspice@siteB=1]" myjob
taskman -Lp P1 -R "rusage[hspice=1]" myjob
taskman -Lp P1 -R "rusage[hspice@clusterA=1]" myjob