4.1 Available Compilers

4.1.1 Compiler versions

In Sisu, all compilers are accessed through the Cray drivers (wrapper scripts) ftn, cc and CC. ftn will launch a Fortran compiler, cc will launch a C compiler and  CC will launch a C++ compiler. No matter which vendor's compiler module is loaded, always use the ftn, cc and CC commands to invoke the compiler.

The Cray driver commands links in the libraries required and they produce code that can be executed on the compute nodes. For more information, see man ftn, man cc or man CC. The compiler arguments (options) to these drivers vary according to which compiler module is loaded. If PrgEnv-cray is loaded then Cray compiler options can be given (either on the command line or in your makefiles). These drivers have also some general options that are valid for all vendor compilers. See the above mentioned man pages for more information. If you compile and link in separate steps use the Cray driver commands also in the linking step and not the linker ld directly.

Sisu login nodes have the Intel Sandy Brigde microarchitecture whereas the compute nodes have the Intel Haswell microarchitecture. The compiler commands produce by default executables for the compute nodes, and in some cases the executables do not work in the login nodes. Please refer to section 4.1.3 on cross compilation if you experience problems in running executables after succesfull compilation or when using GNU autotools (configure; make; make install).

Table 4.1 Compiler suites installed on Sisu:

Compiler suite Programming environment module Man pages User Guides
Cray compilers PrgEnv-cray man crayftn (Fortran)
man craycc (C)
man crayCC (C++)
Fortran reference
C and C++ refence
Gnu Compiler Collection PrgEnv-gnu man gfortran (Fortran)
man gcc (C/C++)
Gfortran manual
Gcc manual
Intel Composer PrgEnv-intel man ifort (Fortran)
man icc (C/C++)
Fortran User and reference
C/C++ User and reference


Command line command module avail will show the compiler suite (module) currently loaded. By swapping the Programming environment module one can change from a compiler suite to another suite. For example:

module swap PrgEnv-cray PrgEnv-gnu
will change from the Cray compiler to the Gnu compiler collection.

Table 4.2 Cray driver commands.
Language (standard)

Compilercommand for Cray, Gnu and Intel compilers

MPI parallel compiler command  How to check the version of a loaded compller
Fortran 77, 95, 2003 ftn ftn ftn -V (for Cray and Intel)
ftn --version (for Gnu)
C cc cc cc -V (for Cray and Intel)
cc --version (for Gnu)
C++ CC CC CC -V (for Cray and Intel)
CC --version (for Gnu)


Any selected compiler suite might have several versions installed. In such a case it is possible to swap the version inside a selected programming environment. The Cray compiler versions have a modulefile naming style cce/version_number, the Gnu compiler collection versions gcc/version_number and the Intel compilers intel/version_number.

For example, swap from a Cray version to another Cray version is done in this way (PrgEnv-cray is the loaded Programming environment):

module swap cce/version_number cce/another_version_number

The command module avail cce will show all available Cray compiler versions (module avail gcc will show all Gnu compiler versions and module avail intel will show all Intel versions).

When compiling a code, it is a good idea to write down all currently loaded modulefiles, i.e., give the command: module list and save the output in a file. This way you can check, what was the environment where your code worked well, if later it will not work properly (for example, after operating system upgrades, or the default version changes of libraries or compilers). If this happens you might load exactly the same modulefiles that you have saved (and see that updates might have not so good effects for your runs).

 

Table 4.3 Compilation examples

Command Result
ftn f_prog.f95 Create a static executable: a.out
ftn -o my_fortran_prog f_routine_1.f95 f_routine_2.f95 Compile Fortran files and link them into a static executable: my_fortran_prog
CC -o my_c++_prog CC_routine.C Will create a static executable: my_c++_prog
cc -c c_routine.c Compile C file, create an object file: c_routine.o
ftn -o my_mixed_c_and fortran_prog f_routine_1.f95 c_routine.o Compile Fortran file and link it with a C object file, create a static executable: my_mixed_c_and_fortran_prog
ftn -c f_routine_2.f95 Compile a Fortran file, create an object file: f_routine_2.o
ftn -o another_mixed_c_and fortran_prog c_routine.o f_routine2.o Link the object files into a static executable: another_mixed_c_and fortran_prog

 

By default Cray drivers will build static executables.  Dynamic builds can be made by the option -dynamic.

ftn -dynamic -o another_mixed_c_and_fortran_prog c_routine.o f_routine2.o
or by defining the environment variable CRAYPE_PE_LINK to have value dynamic.
export CRAYPE_LINK_TYPE=dynamic
ftn -o another_mixed_c_and_fortran_prog c_routine.o f_routine2.o

The command ldd will show the libraries that will be loaded dynamically

ldd another_mixed_c_and_fortran_prog

 

4.1.2 Selecting compiler options

The options available for Cray, Intel, and Gnu compilers can be found on man pages when the corresponding programming environment is loaded, or in the compiler manuals on the Web (see links above on this page).

In the table below some optimization flags are listed for the installed compilers. It is best to start from the safe level and then move up to intermediate or even aggressive, while making sure that the results are correct and the program has a better performance.

Table 4.4 Compiler options for Sisu

Cray GNU Intel
Safe -O1 -O2 -O2 -fp-model precise -fp-model source
(Use all three options. One can also use the options
-fp-model precise -fp-model source
with intermediate and aggressive flags to improve the consistency and reproducibility of floating-point results)
Intermediate -O2 (this is the default) -O3 -O2
Aggressive -O3  (or)  -O3,fp3 -O3 -ffast-math -funroll-loops -march=haswell -mtune=haswell -mavx2 -O3 -opt-prefetch -unroll-aggressive -no-prec-div -fp-model fast=2
Fused multiply-add (FMA)

-hfp1 (disabled)
-hfp2 (default)
-hfp3, fast

-mno-fma (default flag)
-mfma ( enabled )
-no-fma
-fma, (enabled, default flag)

 

Sisu login nodes have the Intel Sandy Brigde microarchitecture whereas the compute nodes have the Intel Haswell microarchitecture. The compiler options for the Haswell architecture are enabled automatically (modulefile craype-haswell is loaded by default).

Haswell microarchitecture has support for fused multiply-add instructions (FMA). Whether using FMA instructios produce any speedup or not, is application and workload dependent.

4.1.3 Cross-compilation

The Haswell instruction set available on Sisu compute nodes is not fully backwards compatible with the Sandy Bridge instruction set available on the compute nodes. It is possible that binaries built to solely use the Haswell instruction set are not executable on the login nodes.

For instance, using an Intel compiler with the -xCORE-AVX2 compilation flag, an attempt to execute the compiled binary on a Sisu login node yields the following error message:

Please verify that both the operating system and the processor support Intel(R) MOVBE, F16C, FMA, BMI, LZCNT and AVX2 instructions.

Some build systems, such as GNU Automake and CMake, separate the configuration step in the compilation process. In the configuration step, a series of small tests is automatically performed on the system by compiling and running small test binaries in order to determine the correct build environment. Therefore, if Haswell instruction set is to be used, such build tools need to be invoked in a cross-compilation mode to avoid incorrect failures in the tests.

GNU Automake: Invoke configure on cross-compilation mode by using the --host specifier. For instance

./configure --host=x86_64-unknown-linux-gnu

For more detailed instructions, see GNU Automake documentation on cross-compilation. Another alternative is to switch to the module craype-sandybridge before running ./configure and then switch back to using the module craype-haswell before the actual compilation step.

CMake: Either use a Toolchain file or the command line to set CMAKE_SYSTEM_NAME and the compilers to be used in the compilation. For instance

cmake /path/to/src -DCMAKE_SYSTEM_NAME=xc30 -DCMAKE_C_COMPILER=cc

For more detailed instructions, see CMake documentation on cross-compilation.

4.1.4 Some Cray compiler options

The Cray compiler enables automatically the recognition of OpenMP directives and no specific compiler option is needed. Position independent code is generated by the options -h pic or -h PIC. To see listings showing which optimizations were performed where in the code, try -r d (Fortran compiler) -h list=m (C/C++ compiler). To see listings showing which potential optimizations were not performed, try -h negmsgs. These might be useful when trying to make changes to your source code. By default, no runtime checks are performed. It is possible to enable various runtime checks. For Fortran this is done with option -R, that has five sub-options (see more information on the man page). The C compiler has an option -h bounds that provides checking of pointer and array references at runtime. These checks might be useful for debugging purposes.

It is possible to enhance the Cray C/C++ compiler compatibility with the Gnu C/C++ Extensions by giving the option -h gnu.

By giving the option -default64 to the ftn Fortran driver it passes -i8 and -r8 options to the Cray compiler. This option also links in the appropriate 64-bit MPI or SHMEM library. So with the Cray compiler MPI and  SHMEM libraries have support for 32bit and 64bit integers. This functionality is not available for the Gnu or Intel compiler suites. When either Gnu or Intel suite is loaded only 32bit integers are accepted by MPI or SHMEM routines.

When the Cray compiler gives messages like error messages, use command explain message-ID to display explanations of the message. For example for Fortran compiler message ID: ftn-100 :

explain ftn-100

And for C/C++ compiler, the message ID:  CC-17:

explain CC-175

 

    Previous chapter     One level up     Next chapter