6.8 Debugging GPU-programs

Debugging GPU programs can be done in multiple ways. For single GPU programs one can use either cuda-gdb or the Nsight IDE. When using these for debugging device code the debugger needs to run on a node that has a GPU in it.

cuda-gdb behaves very similarly to the regular GNU debugger, it works from a command line so no need to have an x11 connection to the compute node it is running on. To run cuda-gdb you need to use an interactive session either using salloc or srun however no additional flags apart from the regular GPU related ones need to be passed to slurm. More information about cuda-gdb: http://docs.nvidia.com/cuda/cuda-gdb/index.html

The nsight IDE is a full-fledged integrated development environment for GPU programs that include a debugging feature for both CPU and GPU code. To run the Nsight IDE for debugging you need to forward the x11 session from the compute node to your local machine. An example of how to run nsight:

srun -n 1 -N 1 --gres=gpu:k80:1 --x11=first -p gpu nsight

Note that the entire time you are running nsight on a compute node you are occupying the number of GPUs requested even if you are just doing code development. For more information about the nsight IDE: http://docs.nvidia.com/cuda/nsight-eclipse-edition-getting-started-guide/index.html