Submitting Jobs with SSH

Secure Shell (SSH) is a network protocol that provides confidentiality and integrity of data using a secure channel between two networked devices.

About SSH

SSH uses public-key cryptography to authenticate the remote computer and allow the remote computer to authenticate the user, if necessary.

SSH is typically used to log into a remote machine and execute commands, but it also supports tunneling, forwarding arbitrary TCP ports and X11 connections. SSH uses a client-server protocol.

SSH uses private/public key pairs to log into another host. Users no longer have to supply a password every time they log on to a remote host.

SSH is used when running any of the following:

  • Remote log on to a lightly loaded host (lslogin)

  • An interactive job (bsub -IS | -ISp | ISs)

  • An interactive X-window job with X11 forwarding (bsub -XF)

  • An interactive X-window job, without X11 forwarding (bsub -IX)

  • An externally submitted job (esub)

X-Window job options

Depending on your requirements for X-Window jobs, you can choose either bsub -XF (recommended) or bsub -IX. Both options encrypt the X-Server and X-Clients.


Mode

Benefits

Drawbacks

bsub -XF (X11 forwarding): Recommended

  • Any password required can be typed in when needed.

  • Does not require the X-Server host to have the SSH daemon installed.

  • The user must enable X11 forwarding in the client.

  • Submission and execution hosts must be UNIX.

bsub -IX (interactive X-window)

  • The execution host contacts the X-Server host directly (no user steps required).

  • Hosts can be any OS that OpenSSH supports.

  • Requires the SSH daemon installed on the X-Server host.

  • Must use private keys with no passwords set.


Scope


Table 1. SSH X11 forwarding (-XF)

Applicability

Details

Dependencies

  • OpenSSH 3.9p1 and up is supported.

    OpenSSL 0.9.7a and up is supported.

  • You must have SSH correctly installed on all hosts in the cluster.

  • You must use an SSH client to log on to the submission host from the display host.

  • You must install and run the X-Server program on the display host.

Operating system

  • Only UNIX for submission and execution hosts. The display host can be any operating system.

Limitations

  • You cannot run with bsub -K, -IX, or -r.

  • You cannot bmod a job submitted with X11 forwarding.

  • Cannot be used with job arrays, job chunks, or user account mapping.

  • Jobs submitted with X11 forwarding cannot be checked or modified by esubs.

  • Can only run on UNIX hosts (submission and execution hosts).



Table 2. Interactive X-window without X11 forwarding (-IX)

Applicability

Details

Dependencies

  • You must have OpenSSH correctly installed on all hosts in the cluster.

  • You must generate public/private key pairs and add the content of the public key to the authorized_keys file on remote hosts. For more information, refer to your SSH documentation.

  • For X-window jobs:

    • You must set the DISPLAY environment variable to X-serverHost:0.0, where X-serverHost is the name of the X-window server. Ensure the X-server can access itself. Run, for example, xhost +localhost.

Operating system

  • Any OS that also supports OpenSSH.

Limitations

  • Cannot be used with job arrays or job chunks.

  • Private user keys must have no password set.

  • You cannot run with -K , -r, or -XF.


Configuration to enable SSH

No LSF configuration is needed to enable SSH X11 forwarding.

Remote log on to a lightly loaded host (lslogin):


Configuration file

Level

Syntax

Behavior

lsf.conf

System

LSF_LSLOGIN_SSH=Y | y

A user with SSH configured can log on to a remote host without providing a password.

All communication between local and remote hosts is encrypted.


Configuration to modify SSH (X11 forwarding)


Configuration file

Level

Syntax

Behavior

lsf.conf

System

LSB_SSH_XFORWARD_CMD

For X11 forwarding, you can modify the default value with an SSH command (full PATH and options allowed).


SSH commands

Commands to submit


Command

Behavior

bsub -IS

Submits a batch interactive job under a secure shell (ssh).

bsub -ISp

Submits a batch interactive job under a secure shell and creates a pseudo-terminal when the job starts.

bsub -ISs

Submits a batch interactive job under a secure shell and creates a pseudo-terminal with shell mode support when the job starts.

Use for interactive shells or applications that redefine the CTRL-C and CTRL-Z keys (for example, jove).

bsub -IX

Submits an interactive X-window job., secured using SSH.

bsub -XF

Submits a job with SSH X11 forwarding.

bsub -XF -I

Submits an interactive job with SSH X11 forwarding. The session displays throughout the job lifecycle.


Commands to monitor


Command

Behavior

netstat -an

Displays all active TCP connections and the TCP and UDP ports on which the computer is listening.

bjobs -l

Displays job information, including any jobs submitted with SSH X11 forwarding.

bhist -l

Displays historical job information, including any jobs submitted with SSH X11 forwarding.


Troubleshoot SSH X11 forwarding (-XF)

SSH X11 forwarding must already working outside LSF.

Enable the following flags in lsf.conf:
  • LSF_NIOS_DEBUG=1

  • LSF_LOG_MASK="LC_TRACE"

Troubleshoot SSH (-IX)

Use the SSH command on the job execution host to connect it securely with the job submission host.

If the host fails to connect, you can perform the following steps to troubleshoot.

  1. Check the SSH version on both hosts.

    If the hosts have different SSH versions, a message displays identifying a protocol version mismatch.

  2. Check that public and private key pairs are correctly configured.

    More information on configuring key pairs is here: http://sial.org/howto/openssh/publickey-auth/.

  3. Check the domain name.

    $ ssh –f –L 6000:localhost:6000 domain_name.example.com date

    $ ssh –f –L 6000:localhost:6000 domain_name date

    If these commands return errors, troubleshoot the domain name with the error information returned.

The execution host should connect without passwords and pass phrases.

$ ssh sahpia03
$ ssh sahpia03.example.com