6.9 Using Taito-GPU's graphics cards for rendering

Graphics oriented applications can utilize Taito-GPU's graphics cards for rendering. Interactive session, rendered on Taito-GPU, is streamed in real time to your local PC via an extra ssh connection. To use this remote graphics method, you need to install a VNC client, TurboVNC, to your local computer.

 

6.9.1 TurboVNC installation and VNC initialization

Download and install TurboVNC to your local PC from

http://sourceforge.net/projects/turbovnc/files/

Create password for Taito-GPU's VNC-server as follows. Login to taito-gpu.csc.fi by using your Taito username and password. When you start VNC-server on Taito-GPU for the first time, you'll be prompted to create a VNC-password (8 characters)

module load vgl
vncserver

When asked "Would you like to enter a view-only password (y/n)?", answer n.

To optimize your default virtual Taito-GPU desktop, edit your .vnc/xstartup.turbovnc file in your Taito-GPU home directory, to include these two lines only

#!/bin/sh
exec openbox-session

To open a terminal on this simple desktop, right click with mouse in the desktop area. To move between windows, click with the middle mouse button.

 

6.9.2 Opening VNC connection to Taito-GPU

The example SLURM batch script gpu_job_reservation.sh, on the bottom of this page, reserves the resources for your Taito-GPU job, prepares the allocated graphics node for rendering, and evokes a VNC server on that node. Copy the script to your Taito-GPU home directory, fill in your email address, edit the job resource reservation parameters, and run the script by issuing

sbatch gpu_job_reservation.sh

Note that the script only works in Taito-GPU, not in Taito.

When your job has been granted resources, the script sends you email how to establish a secure connection to the particular node and port where the job is running. Once established, this extra ssh tunnel is used by Taito-GPU's VNC-server, to securely transfer the image stream of your interactive session to your local PC's TurboVNC app. You will be asked both your Taito-GPU password (same as Taito) and your VNC password (see TurboVNC installation and VNC initialization, above).

Because only the image stream is forwarded, and not the rendering commands, you only need a simple ssh client for the connection. Local X-server and X11 forwarding are not necessary. Even Windows systems do not need NoMachine to use graphical interfaces on Taito-GPU, PuTTY is enough.

After you have established the secure VNC connection to taito-gpu.csc.fi by following the instructions given in the notification email, "Taito-GPU virtual desktop" window opens on your local screen. To run applications in this desktop window, right click with the mouse in the desktop area, and choose Terminal from the pull-down menu. In the terminal, start your application with command vglrun. This evokes VirtualGL middleware, that directs rendering to Taito-GPU's graphics cards. For example, to run VMD, type in the terminal window

module load vmd
vglrun vmd

Note that this remote rendering method is not optimal for ParaView. To run ParaView with several CPUs and GPUs simultaneously, please see the ParaView instructions. For VisIt, use the specific command line provided in the VisIt instructions.

 

6.9.3 Parameters of the script gpu_job_reservation.sh

MAILTO

The email notification address is defined by the MAILTO parameter in the script. Remember to fill in your email address using quotation marks.

If you do not enable email notification, or are unable to read email, you can find out the node and port, allocated to your job, as follows:
Check that your job is running, by issuing

squeue -l -u $USER

When it has started, say job number 12345678, a file job_reservation_12345678.txt is created, which includes the node and display numbers. Type

less job_reservation_12345678.txt

(hit "q" to quit reading) and check the following line: Desktop 'TurboVNC: g43:2 (username)' started on display g43:2. In this example the node is g43, the display is 2, and the port is 5900 + 2 = 5902.

window_size

TurboVNC opens a window on your PC, for the remote rendering session. The default is 1280x800. Edit the script to change this "Taito-GPU virtual desktop" window size, to suit your screen resolution.

putty_path

This parameter is relevant to windows systems only. When installing TurboVNC to your PC, vncviewer and PuTTY are also installed. Notification email instructions for windows refer to the default installation path of 64 bit putty.exe. Edit putty_path parameter if your setup is different, so that you can copy-paste the emailed PuTTY command directly to your Command Prompt window. For example if you installed 32 bit TurboVNC, edit /program files/ to /program files (x86)/.

 

Solving errors

bash: vglrun: command not found

Some module load commands remove other modules. If the module you are using does this, run module load vgl after your module load command, before issuing vglrun ....

[VGL] ERROR: Could not open display / extension "GLX" missing on display

If command vglrun ... gives an error about display, try vglrun -d :$GPU_DEVICE_ORDINAL ... instead. In some cases (for example if you reserved more than one gpu - not recommended) you may have to insert an explicite display number (try 0,1,2,3,4) vglrun -d :0 ...

interactive session gets messed up, local port already in use

For windows systems, you need to explicitely fill in your local port number, for connection to Taito-GPU's node. This local port on your PC must be unique. If the suggested default port 5911 is already in use, say you have already one TurboVNC connection running, pick another 59.. number. Change the corresponding display number in the New TurboVNC Connection window accordingly (localhost:..). For linux/mac systems using vncviewer, free local port is chosen automatically.

persistent problems related to particular gpu nodes

If problems occur when using particular nodes, say g51 and g62, these nodes might be temporarily broken. Exclude them in the script

#SBATCH --exclude=g51,g62
 

VNC-password

If you forget your VNC password, or want to change it, delete password file .vnc/passwd in your home directory, and you will be prompted for a new one, when logging onto Taito-GPU's VNC-server (see TurboVNC installation and VNC initialization, above).

 

Example bash script gpu_job_reservation.sh

#!/bin/bash -l
### --- GPU reservation job script, run it in Taito-gpu --- ###
### --- Edit your job reservation details (#SBATCH lines) --- ###
#SBATCH -J gpu_job_reservation
#SBATCH -o gpu_job_reservation_%j.txt
#SBATCH -e gpu_job_reservation_%j.txt  #do not change this line
# Runtime in hours:minutes:seconds, max is 72 hours (gputest 15 min)
#SBATCH -t 00:45:00
# Use gpu partition (or gputest for 15 minutes)
#SBATCH -p gpu
# How many gpu nodes
#SBATCH --nodes=1
# Optional, which graphics card to use (k40 or k80)
# #SBATCH --constraint=k40
# How many cpu cores (k40/80 gpu nodes have 12/24 cpu cores, max for gputest is 2)
#SBATCH -n 6
# How many gpus (usually apps can use just one, k40/k80 node has 2/4 gpus)
#SBATCH --gres=gpu:1
# How much memory per node (k40/80 node has max 32000/258000 MB)
#SBATCH --mem=8000

### --- Check that the following three parameters are OK --- ###
# Your preferred email address ("username@affiliation.fi")
MAILTO="cscuser@csc.fi"
# Your preferred virtual remote desktop window size
window_size="1280x800"
# For Windows systems only: path to your ssh-client putty.exe
# (this is just for cut-paste to work, when using the emailed instructions)
putty_path="c:\program files\turbovnc\putty.exe"



### --- THE REST OF THE SCRIPT SHOULD NOT BE CHANGED --- ###
module purge
module load vgl

MSGDIR=${HOME}/remotevis
mkdir -p $MSGDIR
MSGFILE=${MSGDIR}/notification.txt

source /etc/slurm/startx
sleep 1
vncserver -geometry ${window_size}
sleep 1

line=`grep "started on display" gpu_job_reservation_${SLURM_JOB_ID}.txt | awk '{ print $NF }'`
dispnumber=`grep "started on display" gpu_job_reservation_${SLURM_JOB_ID}.txt | awk '{ print $NF }' | cut -d: -f 2`
nodename=`grep "started on display" gpu_job_reservation_${SLURM_JOB_ID}.txt | awk '{ print $NF }' | cut -d: -f 1`
portnumber=$((5900 + dispnumber))
cat >$MSGFILE <<END_MESSAGE1
To: ${MAILTO}
From: ${USER}@taito.csc.fi
Subject: ${SLURM_JOB_NAME} ${SLURM_JOB_ID}

Your GPU job ${SLURM_JOB_ID} is running.
Open a secure ssh connection to Taito-gpu's node ${nodename}, port ${portnumber}

---in Linux/Mac---
vncviewer -via ${USER}@taito-gpu.csc.fi ${line}

---in Windows---
from Windows command prompt
"${putty_path}" -L 5911:${nodename}:${portnumber} ${USER}@taito-gpu.csc.fi

Start TurboVNC on your local PC. When "New TurboVNC Connection" window opens, type
localhost:11

-------------

TurboVNC "Taito desktop" window opens on your local screen. Right-click the desktop to open a terminal.
Remember to start your application with command vglrun, to direct your graphics to a GPU card.

---Example to run VMD---
module load vmd
vglrun vmd


More information
https://research.csc.fi/taito-gpu-graphics
END_MESSAGE1

cat $MSGFILE | ssmtp ${MAILTO}
sleep 50
vncpid=`pgrep -s 0 Xvnc`
while [ -e /proc/$vncpid ]; do sleep 1; done