Knowledge Center Contents Previous Next Index |
Job Email and Job File Spooling
Contents
Mail Notification When a Job Starts
When a batch job completes or exits, LSF by default sends a job report by electronic mail to the submitting user account. The report includes the following information:
- Standard output (
stdout
) of the job- Standard error (
stderr
) of the job- LSF job information such as CPU, process and memory usage
The output from
stdout
andstderr
are merged together in the order printed, as if the job was run interactively. The default standard input (stdin
) file is the null device. The null device on UNIX is/dev/null
.bsub mail options
-B
Sends email to the job submitter when the job is dispatched and begins running. The default destination for email is defined by LSB_MAILTO in
lsf.conf
.-u user_name
If you want mail sent to another user, use the
-u
user_name
option to thebsub
command. Mail associated with the job will be sent to the named user instead of to the submitting user account.-N
If you want to separate the job report information from the job output, use the
-N
option to specify that the job report information should be sent by email.Output and error file options (-o output_file, -e error_file, -oo output_file, and -eo error_file)
The output file created by the
-o
and-oo
options to thebsub
command normally contains job report information as well as the job output. This information includes the submitting user and host, the execution host, the CPU time (user plus system time) used by the job, and the exit status.If you specify a
-o
output_file
or-oo
output_file
option and do not specify a-e
error_file
or-eo
error_file
option, the standard output and standard error are merged and stored inoutput_file
. You can also specify the standard input file if the job needs to read input fromstdin
.
note:
The file path can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows, including the directory, file name, and expanded values for %J (job_ID
) and %I (index_ID
).The output files specified by the output and error file options are created on the execution host.
See Remote File Access for an example of copying the output file back to the submission host if the job executes on a file system that is not shared between the submission and execution hosts.
Disabling job email
If you do not want job output to be sent by mail, specify
stdout
andstderr
as the files for the output and error options (-o
,-oo
,-e
, and-eo)
. For example, the following command directsstderr
andstdout
to file named/tmp/job_out
, and no email is sent.bsub -o /tmp/job_out sleep 5On UNIX, If you want no job output or email at all, specify
/dev/null
as the output file:bsub -o /dev/null sleep 5Example
The following example submits
myjob
to the night queue:
bsub -q night -i job_in -o job_out -e job_err myjob
The job reads its input from file
job_in
. Standard output is stored in filejob_out
, and standard error is stored in filejob_err
.Size of job email
Some batch jobs can create large amounts of output. To prevent large job output files from interfering with your mail system, you can use the LSB_MAILSIZE_LIMIT parameter in
lsf.conf
to limit the size of the email containing the job output information.By default, LSB_MAILSIZE_LIMIT is not enabled-no limit is set on size of batch job output email.
If the size of the job output email exceeds LSB_MAILSIZE_LIMIT, the output is saved to a file under JOB_SPOOL_DIR, or the default job output directory if JOB_SPOOL_DIR is undefined. The email informs users where the job output is located.
If the
-o
or-oo
option ofbsub
is used, the size of the job output is not checked against LSB_MAILSIZE_LIMIT.LSB_MAILSIZE environment variable
LSF sets LSB_MAILSIZE to the approximate size in KB of the email containing job output information, allowing a custom mail program to intercept output that is larger than desired. If you use the LSB_MAILPROG parameter to specify the custom mail program that can make use of the LSB_MAILSIZE environment variable, it is not necessary to configure LSB_MAILSIZE_LIMIT.
LSB_MAILSIZE is not recognized by the LSF default mail program. To prevent large job output files from interfering with your mail system, use LSB_MAILSIZE_LIMIT to explicitly set the maximum size in KB of the email containing the job information.
LSB_MAILSIZE values
The LSB_MAILSIZE environment variable can take the following values:
- A positive integer: if the output is being sent by email, LSB_MAILSIZE is set to the estimated mail size in KB.
-1
:if the output fails or cannot be read, LSB_MAILSIZE is set to -1 and the output is sent by email using LSB_MAILPROG if specified inlsf.conf
.- Undefined: If you use the output or error options (
-o
,-oo
,-e
, or-eo)
ofbsub
, the output is redirected to an output file. Because the output is not sent by email in this case, LSB_MAILSIZE is not used and LSB_MAILPROG is not called.If the
-N
option is used with the output or error options ofbsub
, LSB_MAILSIZE is not set.Directory for job output
The output and error options (
-o
,-oo
,-e
, and-eo
) of thebsub
andbmod
commands can accept a file name or directory path. LSF creates the standard output and standard error files in this directory. If you specify only a directory path, job output and error files are created with unique names based on the job ID so that you can use a single directory for all job output, rather than having to create separate output directories for each job.
note:
The directory path can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows.Specifying a directory for job output
Make the final character in the path a slash (/) on UNIX, or a double backslash (\\) on Windows. If you omit the trailing slash or backslash characters, LSF treats the specification as a file name.
If the specified directory does not exist, LSF creates it on the execution host when it creates the standard error and standard output files.
By default, the output files have the following format:
Standard output
output_directory/job_ID.out
Standard error
error_directory/job_ID.err
Example
The following command creates the directory
/usr/share/lsf_out
if it does not exist, and creates the standard output filejob_ID
.out
in this directory when the job completes:bsub -o /usr/share/lsf_out/ myjob
The following command creates the directory
C:\lsf\work\lsf_err
if it does not exist, and creates the standard error filejob_ID
.err
in this directory when the job completes:bsub -e C:\lsf\work\lsf_err\\ myjob
For more information
See the
Platform LSF Configuration Reference
for information about the LSB_MAILSIZE environment variable and the LSB_MAILTO, LSB_MAILSIZE_LIMIT parameters inlsf.conf
, and JOB_SPOOL_DIR inlsb.params
.File Spooling for Job Input, Output, and Command Files
About job file spooling
LSF enables
spooling
of job input, output, and command files by creating directories and files for buffering input and output for a job. LSF removes these files when the job completes.You can make use of file spooling when submitting jobs with the
-is
and-Zs
options tobsub
. Use similar options inbmod
to modify or cancel the spool file specification for the job. Use the file spooling options if you need to modify or remove the original job input or command files before the job completes. Removing or modifying the original input file does not affect the submitted job.
note:
The file path for spooling job input, output, and command files can contain up to 4094 characters for UNIX and Linux, or up to 255 characters for Windows, including the directory, file name, and expanded values for %J (job_ID
) and %I (index_ID
).File spooling is not supported across MultiClusters.
Specifying job input files
Use the
bsub -i input_file
andbsub -is input_file
commands to get the standard input for the job from the file path name specified byinput_file
. The path can be an absolute path or a relative path to the current working directory. The input file can be any type of file, though it is typically a shell script text file.LSF first checks the execution host to see if the input file exists. If the file exists on the execution host, LSF uses this file as the input file for the job.
If the file does not exist on the execution host, LSF attempts to copy the file from the submission host to the execution host. For the file copy to be successful, you must allow remote copy (
rcp
) access, or you must submit the job from a server host where RES is running. The file is copied from the submission host to a temporary file in the directory specified by the JOB_SPOOL_DIR parameter inlsb.params
, or your$HOME/.lsbatch
directory on the execution host. LSF removes this file when the job completes.The
-is
option ofbsub
spools the input file to the directory specified by the JOB_SPOOL_DIR parameter inlsb.params
, and uses the spooled file as the input file for the job.Use the
bsub -is
command if you need to change the original input file before the job completes. Removing or modifying the original input file does not affect the submitted job.Unless you use
-is
, you can use the special characters%J
and%I
in the name of the input file.%J
is replaced by the job ID.%I
is replaced by the index of the job in the array, if the job is a member of an array, otherwise by 0 (zero). The special characters%J
and%I
are not valid with the-is
option.Specifying a job command file (bsub -Zs)
Use the
bsub -Zs
command to spool a job command file to the directory specified by the JOB_SPOOL_DIR parameter inlsb.params
. LSF uses the spooled file as the command file for the job.Use the
bmod -Zs
command if you need to change the command file after the job has been submitted. Changing the original input file does not affect the submitted job. Usebmod -Zsn
to cancel the last spooled command file and use the original spooled file.The bsub -
Zs
option is not supported for embedded job commands because LSF is unable to determine the first command to be spooled in an embedded job command.About the job spooling directory (JOB_SPOOL_DIR)
If JOB_SPOOL_DIR is specified in
lsb.params
:
- The job input file for
bsub -is
is spooled toJOB_SPOOL_DIR/lsf_indir
. If thelsf_indir
directory does not exist, LSF creates it before spooling the file. LSF removes the spooled file when the job completes.- The job command file for
bsub -Zs
is spooled toJOB_SPOOL_DIR/lsf_cmddir
. If thelsf_cmddir
directory does not exist, LSF creates it before spooling the file. LSF removes the spooled file when the job completes.The JOB_SPOOL_DIR directory should be a shared directory accessible from the master host and the submission host. The directory must be readable and writable by the job submission users.
JOB_SPOOL_DIR can be any valid path up to a maximum length up to 4094 characters on UNIX and Linux or up to 255 characters for Windows.
JOB_SPOOL_DIR must be readable and writable by the job submission user, and it must be shared by the master host and the submission host. If the specified directory is not accessible or does not exist,
bsub -is
cannot write to the default directoryLSB_SHAREDIR/
cluster_name
/lsf_cmddir
and the job will fail.Except for
bsub -is
andbsub -Zs
, if JOB_SPOOL_DIR is not accessible or does not exist, output is spooled to the default job output directory.lsbatch
.For
bsub -is
andbsub -Zs
, JOB_SPOOL_DIR must be readable and writable by the job submission user. If the specified directory is not accessible or does not exist,bsub -is
andbsub -Zs
cannot write to the default directory and the job will fail.If JOB_SPOOL_DIR is not specified in
lsb.params
:
- The job input file for
bsub -is
is spooled toLSB_SHAREDIR/
cluster_name
/lsf_indir
. If thelsf_indir
directory does not exist, LSF creates it before spooling the file. LSF removes the spooled file when the job completes.- The job command file for
bsub -Zs
is spooled toLSB_SHAREDIR/
cluster_name
/lsf_cmddir
. If thelsf_cmddir
directory does not exist, LSF creates it before spooling the file. LSF removes the spooled file when the job completes.If you want to use job file spooling, but do not specify JOB_SPOOL_DIR, the
LSB_SHAREDIR/
cluster_name
directory must be readable and writable by all the job submission users. If your site does not permit this, you must manually createlsf_indir
andlsf_cmddir
directories underLSB_SHAREDIR/
cluster_name
that are readable and writable by all job submission users.Modifying the job input file
Use the
-i
and-is
options ofbmod
to specify a new job input file. The-in
and-isn
options cancel the last job input file modification made with either-i
or-is
.Modifying the job command file
Use the
-Z
and-Zs
options ofbmod
to modify the job command file specification.-Z
modifies a command submitted without spooling, andZs
modifies a spooled command file. The-Zsn
option ofbmod
cancels the last job command file modification made with-Zs
and uses the original spooled command.For more information
See the
Platform LSF Command Reference
for more information about thebsub
andbmod
commands.See the Platform LSF Configuration Reference for more information about the JOB_SPOOL_DIR parameter in
lsb.params
, and the LSF_TMPDIR environment variable.
Platform Computing Inc.
www.platform.com |
Knowledge Center Contents Previous Next Index |