Learn more about Platform products at http://www.platform.com

[ Platform Documentation ] [ Title ] [ Contents ] [ Previous ] [ Next ] [ Index ]



Installing and Upgrading Platform LSF HPC


[ Top ]


Installing Platform LSF HPC

Installing Platform LSF HPC involves the following steps:

  1. Get a Platform LSF HPC license.
  2. Download Platform LSF HPC Packages.
  3. Run lsfinstall.
  4. Run hostsetup to configure host-based resources and set up automatic LSF startup on server hosts.


    Running hostsetup is optional on AIX and Linux. You must run hostsetup on SGI hosts (IRIX, TRIX, and Altix), HP-UX hosts, and Linux QsNet hosts.

ENABLE_HPC_CONFIG

Make sure ENABLE_HPC_CONFIG=Y is specified in install.config to enable Platform LSF HPC features.

Get a Platform LSF HPC license

Before installing Platform LSF HPC, you must get a permanent or demo license key. Contact Platform Computing at license@platform.com to request a license key.

Copy the license key to a file named license.dat in the same directory where you downloaded the LSF HPC distribution tar files.

Download Platform LSF HPC Packages

Use FTP to download the Platform LSF HPC distribution packages.

Access to the Platform FTP site is controlled by login name and password.

The Platform LSF HPC distribution packages are located in /distrib/7.0/.

The Platform LSF Version 7 lsf7Update5_release_notes.html file for information about downloading the LSF HPC distribution packages.

Before installing

MPICH-GM with AFS

If you are planning to use AFS with MPICH-GM and have made any custom changes to your existing AFS or MPICH-GM esub, create a backup of these.

What lsfinstall does

lsb.hosts

For the default host, lsfinstall enables "!" in the MXJ column of the HOSTS section of lsb.hosts. For example:

Begin Host
HOST_NAME MXJ   r1m     pg    ls    tmp  DISPATCH_WINDOW  # Keywords
#hostA     () 3.5/4.5   15/   12/15  0      ()            # Example
default    !    ()      ()    ()     ()     ()            
HPPA11     !    ()      ()    ()     ()     ()            #pset host
End Host

lsb.modules

lsb.resources

For IBM POE jobs, lsfinstall configures the ReservationUsage section in lsb.resources to reserve HPS resources on a per-slot basis.

Resource usage defined in the ReservationUsage section overrides the cluster-wide RESOURCE_RESERVE_PER_SLOT parameter defined in lsb.params if it also exists.

Begin ReservationUsage
RESOURCE           METHOD
adapter_windows    PER_SLOT
ntbl_windows       PER_SLOT
csss               PER_SLOT
css0               PER_SLOT
End ReservationUsage

lsb.queues

Begin Queue
QUEUE_NAME   = rms
PJOB_LIMIT   = 1
PRIORITY     = 30
NICE         = 20
STACKLIMIT   = 5256
DEFAULT_EXTSCHED = RMS[RMS_SNODE]  # LSF will using this scheduling policy if
                                   # -extsched is not defined.
# MANDATORY_EXTSCHED = RMS[RMS_SNODE] # LSF enforces this scheduling policy
RES_REQ = select[rms==1]
DESCRIPTION  = Run RMS jobs only on hosts that have resource 'rms' defined
End Queue


To make the one of the LSF queues the default queue, set DEFAULT_QUEUE in lsb.params.

Use the bqueues -l command to view the queue configuration details. Before using LSF HPC, see the Platform LSF Configuration Reference to understand queue configuration parameters in lsb.queues.

lsf.cluster.cluster_name

lsf.conf

lsf.shared

Defines the following shared resources required by LSF HPC in lsf.shared:

Begin Resource
RESOURCENAME    TYPE    INTERVAL INCREASING  DESCRIPTION       # Keywords
rms             Boolean    ()    ()          (RMS)
pset            Boolean    ()    ()          (PSET)
slurm           Boolean    ()    ()          (SLURM)
cpuset          Boolean    ()    ()          (CPUSET)
mpich_gm        Boolean    ()    ()          (MPICH GM MPI)
lammpi          Boolean    ()    ()          (LAM MPI)
mpichp4         Boolean    ()    ()          (MPICH P4 MPI)
mvapich         Boolean    ()    ()          (Infiniband MPI)
sca_mpimon      Boolean    ()    ()          (SCALI MPI)
ibmmpi          Boolean    ()    ()          (IBM POE MPI)
hpmpi           Boolean    ()    ()          (HP MPI)
sgimpi          Boolean    ()    ()          (SGI MPI)
intelmpi        Boolean    ()    ()          (Intel MPI)
crayxt3         Boolean    ()    ()          (Cray XT3 MPI)
crayx1          Boolean    ()    ()          (Cray X1 MPI)
fluent          Boolean    ()    ()          (fluent availability)
ls_dyna         Boolean    ()    ()          (ls_dyna availability)
nastran         Boolean    ()    ()          (nastran availability)
pvm             Boolean    ()    ()          (pvm availability)
openmp          Boolean    ()    ()          (openmp availability)
ansys           Boolean    ()    ()          (ansys availability)
blast           Boolean    ()    ()          (blast availability)
gaussian        Boolean    ()    ()          (gaussian availability)
lion            Boolean    ()    ()          (lion availability)
scitegic        Boolean    ()    ()          (scitegic availability)
schroedinger    Boolean    ()    ()          (schroedinger availability)
hmmer           Boolean    ()    ()          (hmmer availability)
adapter_windows Numeric    30    N    (free adapter windows on css0 on IBM SP)
ntbl_windows    Numeric    30    N    (free ntbl windows on IBM HPS)
poe             Numeric    30    N    (poe availability)
css0            Numeric    30    N    (free adapter windows on css0 on IBM SP)
csss            Numeric    30    N    (free adapter windows on csss on IBM SP)
dedicated_tasks Numeric    ()    Y    (running dedicated tasks)
ip_tasks        Numeric    ()    Y    (running IP tasks)
us_tasks        Numeric    ()    Y    (running US tasks)
End Resource


You should add the appropriate resource names under the RESOURCES column of the Host section of lsf.cluster.cluster_name.

Run lsfinstall

lsfinstall runs the LSF installation scripts and configuration utilities to install a new Platform LSF cluster or to upgrade LSF from a previous release.


Make sure ENABLE_HPC_CONFIG=Y is specified in install.config to enable Platform LSF HPC features.

To run lsfinstall

  1. Log on to the file server host as root.


    You can run lsfinstall as a non-root user, but to install a fully operational LSF cluster that all users can access, you should install as root.

  2. Download, uncompress, and extract lsf7Update5_lsfinstall.tar.Z to the distribution directory where you downloaded the LSF HPC product distribution tar files.


    Do not extract the Platform LSF HPC distribution files.

  3. Change to the directory lsf7Update5_lsfinstall/.
  4. Read lsf7Update5_lsfinstall/install.config or lsf7Update5_lsfinstall/slave.config and decide which installation variables you need to set.
  5. Edit lsf7Update5_lsfinstall/install.config or lsf7Update5_lsfinstall/slave.config.

    Uncomment any other options you want in the template file, and replace the example values with your own settings.


    The sample values in the install.config and slave.config template files are examples only. They are not default installation values.

  6. Run lsfinstall as root:
    # ./lsfinstall -f install.config
    

If you install as a non-root user

You can install Platform LSF HPC as a non-root user with some limitations. During installation, lsfinstall detects that you are not root. You must choose to configure either a multi-user cluster or a single-user cluster:

Run hostsetup


You must run hostsetup on SGI hosts (IRIX, TRIX, and Altix), HP-UX hosts, and Linux QsNet hosts. Running hostsetup is optional on all other systems.

What hostsetup does

--boot option

Use the --boot="y" option on hostsetup to configure system scripts to automatically start and stop LSF daemons at system startup or shutdown. You must run hostsetup as root to use this option to modify the system scripts. The default is --boot="n".

For complete hostsetup usage, enter hostsetup -h.

To run hostsetup

  1. Log on to each LSF server host as root. Start with the LSF master host.
  2. Run hostsetup on each LSF server host. For example:
    # cd /usr/share/hpc/7.0/install
    # ./hostsetup --top="/usr/share/hpc" --boot="y"
    

Optional configuration

After installation, you can define the following in lsf.conf:

Before using your cluster

After installing LSF and setting up your server hosts:

  1. Log on to the LSF master host as root.
  2. Set your environment:
    • For csh or tcsh:
      % source /LSF_TOP/conf/cshrc.lsf
      
    • For sh, ksh, or bash:
      $ . /LSF_TOP/conf/profile.lsf
      
  3. Use lsfstartup to start the cluster.


    For a large cluster, where cluster management software exists, you should use /etc/init.d lsf start instead of lsfstartup.

  4. Follow the steps in lsf7.0_lsfinstall/lsf_quick_admin.html to verify that your cluster is operating correctly.
  5. Have users run one of the shell environment files to switch their environment to the new cluster.

Enable Platform LSF HPC for users


After testing your cluster, be sure all users include LSF_CONFDIR/cshrc.lsf or LSF_CONFDIR/profile.lsf in their .cshrc or .profile. Follow the steps in lsf7.0_lsfinstall/lsf_quick_admin.html for using LSF_CONFDIR/cshrc.lsf and LSF_CONFDIR/profile.lsf to set up the Platform HPC environment for users.

After the new cluster is up and running, users can start submitting jobs to it.

[ Top ]


Upgrading Platform LSF HPC

CAUTION


If your cluster was installed or upgraded with lsfsetup, DO NOT use these steps. Before upgrading Platform LSF HPC, upgrade your cluster to at least Platform LSF Version 6.0.

Before upgrading

  1. Back up your existing LSF_CONFDIR, LSB_CONFDIR, and LSB_SHAREDIR according to the procedures at your site.
  2. Get an LSF HPC Version 7 license and create a license file (license.dat).
  3. Inactivate all queues to make sure that no new jobs will be dispatched during the upgrade.


    After upgrading, remember to activate the queues again so pending jobs can be dispatched.

    • To inactivate all LSF queues, use the following command:
      badmin qinact all
      
    • To reactivate all LSF queues after upgrading, use the following command:
      badmin qact all
      
  4. For SGI cpuset hosts, make sure all running jobs are done (all queues are drained of running jobs).

What lsfinstall does for upgrade

Configuration file backup

lsfinstall backs up the following configuration files for your current installation in LSF_CONFDIR:

lsb.queues

LSB_SUB_COMMANDNAME (lsf.conf)


If LSB_SUB_COMMANDNAME=N is already defined in lsf.conf, lsfinstall does not change this parameter; you must manually set it to LSB_SUB_COMMANDNAME=Y to enable the LSF_SUB_COMMANDLINE environment variable required by esub.

SGI cpuset host upgrade

For SGI cpuset hosts, lsfinstall updates the following files:

Reusing install.config from your existing installation

You can reuse the install.config file from your existing installation to specify your installation options. The install.config file containing the options you specified for your original installation is located in LSF_TOP/lsf_version/install/.

If you change install.config to add new hosts in LSF_ADD_SERVERS and LSF_ADD_CLIENTS, or new LSF administrators in LSF_ADMINS, lsfinstall creates a new lsf.cluster.cluster_name file.

Run lsfinstall to upgrade

Make sure the following install.config variables are set for upgrade:

Migrate from LSF to LSF HPC


To migrate an existing Platform LSF Version 7 cluster to Platform LSF HPC. comment out LSF_TARDIR and make sure that no distribution tar files are in the directory where you run lsfinstall.

To run lsfinstall

  1. Log on to the file server host as root.
  2. Download, uncompress, and extract lsf7Update5_lsfinstall.tar.Z to the distribution directory where you downloaded the LSF HPC product distribution tar files.
  3. Change to the directory lsf7Update5_lsfinstall/.
  4. Read lsf7Update5_lsfinstall/install.config or lsf7Update5_lsfinstall/slave.config and decide which installation variables you need to set.
  5. Edit lsf7Update5_lsfinstall/install.config or lsf7Update5_lsfinstall/slave.config.


    To enable configuration of Platform LSF HPC feaures, specifiy ENABLE_HPC_CONFIG=Y in install.config.

  6. Run lsfinstall as root:
    # ./lsfinstall -f install.config
    

Run hostsetup


Running hostsetup is optional on AIX and Linux. You must run hostsetup on SGI hosts (IRIX, TRIX, and Altix) and on HP-UX hosts.

What hostsetup does

--boot option

Use the --boot="y" option on hostsetup to configure system scripts to automatically start and stop LSF HPC daemons at system startup or shutdown. You must run hostsetup as root to use this option to modify the system scripts. The default is --boot="n".

For complete hostsetup usage, enter hostsetup -h.

To run hostsetup

  1. Log on to each LSF server host as root. Start with the LSF master host.
  2. Run hostsetup on each LSF server host. For example:
    # cd /usr/share/hpc/7.0/install
    # ./hostsetup --top="/usr/share/hpc" --boot="y"
    

After upgrading

  1. Log on to the LSF master host as root.
  2. Set your environment:
    • For csh or tcsh:
      % source /LSF_TOP/conf/cshrc.lsf
      
    • For sh, ksh, or bash:
      # . /LSF_TOP/conf/profile.lsf
      
  3. Follow the steps in lsf7Update5_lsfinstall/lsf_quick_admin.html to update your license.
  4. Use the following commands to shut down the old LSF daemons:
    # badmin hshutdown all
    # lsadmin resshutdown all
    # lsadmin limshutdown all
    
  5. Use the following commands to start Platform LSF HPC using the upgraded daemons:
    # lsadmin limstartup all
    # lsadmin resstartup all
    # badmin hstartup all
    
  6. Follow the steps in lsf7.0_lsfinstall/lsf_quick_admin.html to verify that your upgraded cluster is operating correctly.
  7. Use the following command to reactivate all LSF HPC queues after upgrading:
    # badmin qact all
    
  8. Have users run one of the shell environment files to switch their environment to the new cluster.

Enable Platform LSF HPC for users


After testing your cluster, be sure all users include LSF_CONFDIR/cshrc.lsf or LSF_CONFDIR/profile.lsf in their .cshrc or .profile. Follow the steps in lsf7.0_lsfinstall/lsf_quick_admin.html for using LSF_CONFDIR/cshrc.lsf and LSF_CONFDIR/profile.lsf to set up the Platform HPC environment for users.

After your cluster is up and running, users can start submitting jobs to it.

[ Top ]


[ Platform Documentation ] [ Title ] [ Contents ] [ Previous ] [ Next ] [ Index ]


      Date Modified: March 13, 2009
Platform Computing: www.platform.com

Platform Support: support@platform.com
Platform Information Development: doc@platform.com

Copyright © 1994-2009 Platform Computing Corporation. All rights reserved.