6.5 Running GPU-programs

6.5.1 Introduction

It is always recommended to use SLURM batch job file for running GPU specific jobs. However, for quick tests also srun command would be acceptable. When running GPU applications the batch queue (or partition) in concern need to be either "gpu", "gputest", "gpulong" Which partition used is set using

-p queue_name

What type of GPU used is set using a generic resource (GRES) flag:

--gres=gpu:type:n

Where type is the type of GPU requested per compute node, currently valid values are k40 or k80 or p100 and n is the number of GPUs to reserve per node. On the K40 nodes one can reserve up to 2 GPUs, while on the K80 and P100 nodes one can reserve up to 4. The gpu and gpulong partition is intended for production runs while gputest is intended for short (less than 15 min) test and development runs.

To request 2 K40 GPUs use --gres=gpu:k40:2 or to request 4 P100 GPUs use --gres=gpu:p100:4

6.5.2 Running under GNU environment on one GPU

Here is a valid script for running under GNU environment that will default any GPU available:

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -p gpu
#SBATCH -t 00:05:00
#SBATCH -J gpu_job
#SBATCH -o gpu_job.out.%j
#SBATCH -e gpu_job.err.%j
#SBATCH --gres=gpu:1
#SBATCH

module purge
module load gcc cuda
module list

srun ./your_binary

6.5.3 Running under PGI environment on one GPU

The batch job script is very similar as in the GPU environment, the only difference is that different module needs to be loaded. And executable name is also different.

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -p gpu
#SBATCH -t 00:05:00
#SBATCH -J gpu_job
#SBATCH -o gpu_job.out.%j
#SBATCH -e gpu_job.err.%j
#SBATCH --gres=gpu:1
#SBATCH

module purge
module load pgi cuda/7.5
module list

srun ./your_binary

6.5.4 Running under GNU environment on multiple GPUs

Here is a valid script for running under GNU environment, on 2 GPUs on a K40 node:

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -p gpu
#SBATCH -t 00:05:00
#SBATCH -J gpu_job
#SBATCH -o gpu_job.out.%j
#SBATCH -e gpu_job.err.%j
#SBATCH --gres=gpu:k40:2
#SBATCH

module purge
module load gcc cuda
module list

srun ./your_binary

Here is a valid script for running under GNU environment, on 4 GPUs on K80 a node:

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -p gpu
#SBATCH -t 00:05:00
#SBATCH -J gpu_job
#SBATCH -o gpu_job.out.%j
#SBATCH -e gpu_job.err.%j
#SBATCH --gres=gpu:k80:4
#SBATCH

module purge
module load gcc cuda
module list

srun ./your_binary

6.5.5 Using the SSD scratch space

Each P100 node is equipped with 800GB of fast SSD storage (Sata SSDs) intended as a scratch space. While the K40 and K80 nodes have 500 GB and 850 GB of space respectively consisting of regular hard drives. This space is placed under the $TMPDIR directory and in that directory there will be a folder with the name of the slurm job id, it is this folder that should be used as a scratch space so to access it easily use $TMPDIR. The space is local to each compute node meaning that it is shared among all the users of that node and in multi node jobs each node only has access to its own local scratch space and this cannot be accessed from other compute nodes.
To use this space the user needs to move data there in the beginning of the job scrip and any data the user wants to retain needs to be transferred from the scratch space before the job finishes, once the job ends the files on the SSD space are removed.

Data can easily be moved to the $TMPDIR using the sbcastsbacast  command. Here is a simple scrip that copies some data from $WRKDIR to $TMPDIR before the application is launched.

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -p gpu
#SBATCH -t 00:05:00
#SBATCH -J gpu_job
#SBATCH -o gpu_job.out.%j
#SBATCH -e gpu_job.err.%j
#SBATCH --gres=gpu:p100:4
#SBATCH

module purge
module load gcc cuda
module list

sbcast $WRKDIR/your_file.csv $TMPDIR/your_file.csv
srun ./your_binary

 

Previous chapter     One level up     Next chapter