[ Platform Documentation ] [ Title ] [ Contents ] [ Previous ] [ Next ] [ Index ]
[ Top ]
About Platform LSF and MVAPICH
MVAPICH is an open-source product developed in the Department of Computer and Information Science, The Ohio State University. MVAPICH is MPI-1 over VAPI for InfiniBand. It is an MPI-1 implementation on Verbs Level Interface (VAPI), developed by Mellanox Technologies. The implementation is based on MPICH and MVICH.
The LSF MVAPICH MPI integration is based on the LSF generic PJL framework. It supports the following MVAPICH variations:
Requirements
- The latest release is MVAPICH 0.9.4 (includes MPICH 1.2.6). or later
You should upgrade all your hosts to the same version of MVAPICH.
Assumptions and limitations
- MVAPICH is installed and configured correctly
- The user's current working directory is part of a shared file system reachable by all hosts
- The directory specified by the MVAPICH_HOME variable is accessible by the same path on all hosts
Glossary
(Message Passing Interface) A message passing standard. It defines a message passing API useful for parallel and distributed applications.
A portable implementation of the MPI standard.
(Parallel Application Manager) The supervisor of any parallel job.
(Parallel Job Launcher) Any executable script or binary capable of starting parallel tasks on all hosts assigned for a parallel job.
(Remote Execution Server) An LSF daemon residing on each host. It monitors and manages all LSF tasks on the host.
(TaskStarter) An executable responsible for starting a task on the local host and reporting the process ID and host name to the PAM.
For more information
- See the Mathematics and Computer Science Division (MCS) of Argonne National Laboratory (ANL) MPICH Web page at
www-unix.mcs.anl.gov/mpi/mpich/
for more information about MPICH.- MVAPICH HOME:
nowlab.cis.ohio-state.edu/projects/mpi-iba/
- ROCKS HOME:
www.rocksclusters.org/Rocks/
- Topspin (now Cisco):
http://cisco.com/en/US/products/index.html
- IBRIX roll:
http://www.rocksclusters.org/Rocks/
Files installed by lsfinstall
During installation,
lsfinstall
copies these files to the following directories:
Resources and parameters configured by lsfinstall
- External resources in
lsf.shared
:Begin Resource RESOURCE_NAME TYPE INTERVAL INCREASING DESCRIPTION ... mvapich Boolean () () (Infiniband MPI) ... End ResourcesThe
mvapich
Boolean resource is used for mapping hosts with MVAPICH available.
You should add themvapich
resource name under the RESOURCES column of the Host section oflsf.cluster.
cluster_name.
- Parameter to
lsf.conf
:LSB_SUB_COMMANDNAME=y[ Top ]
Configuring LSF to Work with MVAPICH
esub.mvapich script
Modify the
esub.mvapich
inLSF_SERVERDIR
to set MVAPICH_START_CMD.to one ofssh
,rsh
, ormpd
. The default value isssh
.mvapich_wrapper script
Modify the
mvapich_wrapper
script inLSF_BINDIR
to set MVAPICH_HOME. The defaults are:
- Topspin/Cisco MPI:
MVAPICH_HOME="/usr/local/topspin
- IBRIX Roll MPI:
MVAPICH_HOME="/opt/mpich/infiniband/gnu"
- Generic MVAPICH: defined by your site. For example:
MVAPICH_HOME="/opt/mvapich"
Make sure the mpirun_rsh/ssh/mpd command is under MVAPICH_HOME/bin.
[ Top ]
Submitting MVAPICH Jobs
bsub command
Use
bsub -a mvapich
to submit jobs:If the starting command is
mpd
, you must submit your MVAPICH jobs as exclusive jobs (bsub -x
).bsub -a mvapich -n
number_cpusmpirun.lsf
[-pam "
pam_options"
] [mpi_options] job [job_options]
-a mvapich
tellsesub
the job is an MVAPICH job and invokesesub.mvapich
.-n
number_cpus specifies the number of processors required to run the job
mpirun.lsf
reads the environment variable LSF_PJL_TYPE=mvapich set byesub.
mvapich
, and generates the appropriatepam
command line to invoke MVAPICH as the PJLFor example:
% bsub -a mvapich -n 3 mpirun.lsf /examples/cpi
A job named
cpi
will be dispatched and run on 3 CPUs in parallel.Task geometry with MVAPICH jobs
MVAPICH supports the LSF task geometry feature except in MPD mode. When running in MPD mode, the order of the hosts specified in the machine file is not honored:
Submitting a job with a job script
A wrapper script is often used to call MVAPICH. You can submit a job using a job script as an embedded script or directly as a job, for example:
% bsub -a mvapich -n 4 < embedded_jobscript
% bsub -amvapich
-n 4
jobscriptYour job script must use
mpirun.lsf
in place of thempirun
command.For more information
- See Running Parallel Jobs for information about generic PJL wrapper script components
- See the Platform LSF Command Reference for information about the
bsub
command- See Administering Platform LSF for information about submitting jobs with job scripts
[ Top ]
[ Platform Documentation ] [ Title ] [ Contents ] [ Previous ] [ Next ] [ Index ]
Date Modified: January 10, 2011
Platform Computing: www.platform.com
Platform Support: support@platform.com
Platform Information Development: doc@platform.com
Copyright © 1994-2011 Platform Computing Corporation. All rights reserved.