1.1 Taito supercluster

1.1.1 User policy

The Taito supercluster (taito.csc.fi) is intended for serial and medium sized parallel tasks as well as jobs that require a lot of memory. Researchers that want to use Taito should 1) register as CSC users and then 2) apply for a computing project. This registration process is described in the chapters 1.2.1 and 1.2.2.2 of the CSC computing environment user guide.

A computing project at CSC has a common computing quota that can be extended by application. Use of Taito or any other server will consume the computing quota granted to the project. One core hour in Taito consumes 2 billing units from the computing quota of the project.

The Taito users are allowed to submit up to 1024 simultaneous batch jobs to be executed. The maximum size of a single job is at most 448 compute cores using 672 CPUs.

Table 1.1 Available batch jobs queues in supercluster taito.csc.fi.

Queue Number of cores Maximum run time
serial (default) 24 (one node) 3 days
parallel 672 (28 nodes) 3 days
longrun 24 (one node) 14 days
test 32  / 48 (two nodes) 30 min
hugemem 40 (one haswell hugemem node) 7 days/14days

 

In Taito you don't need to sent the scaling test results of your parallel code to CSC. However, you should still make sure that you are using the resources efficiently i.e. that your code - with the used input - does scale to the selected number of cores. The rule of thumb is that when you double the number of cores, the job should to run at least 1.5 times faster. If it doesn't, you should use less cores. Note that scaling depends on the input (model system) as well as the used code, so you may need to test separately for scaling with the same code for different model systems. If you are unsure, contact CSC Service Desk.

1.1.2 Hardware

Taito (taito.csc.fi) is a 16 cabinet HP cluster based on commodity off-the-shelf building blocks. The theoretical peak performance of the cluster, calculated on the aggregate performance of the computing nodes, is about 600 TFlop/s. 

Sandy bridge nodes

As a preparation for the installallation of the new cluster environment, the older part of Taito, consisting of 576 Sandy Bridge nodes, was removed in January 2019.

Haswell nodes

The Haswell processors comprise several components: twelve cores with individual L1 and L2 caches, an integrated memory controller, three QPI links, and an L3 cache shared within the socket. The processor supports several instructions sets, most notably the Advanced Vector Extensions 2 (AVX2) instruction set; however, older instructions sets are still supported. Each Haswell core has dedicated 32KB of L1 cache, and 768 KB of L2 cache. The L3 cache is shared among the processors, and its size is 30 MB. Most of the Haswell nodes (397) have 8 slots of 16GB DDR4 DIMMs, operating at 2133 MHz, for a total of 128GB per compute node. This means that there are 5,3 GB of memory available per core. There are also ten Haswell-equipped nodes with 16 modules of 16 GB 2133 MHz DIMMs, that is  256 GB of DDR4 memory per node, and 10,7 GB per core.

Hugemem nodes

For jobs requiring very large memory, Taito includes six hugemem nodes each having 1,5 TB of memory:

  • two HP Proliant DL560 nodes, with 2.7Ghz Sandy Bridge processors  with 32 cores (four eight node sockets) and 2 TB of  local temporary storage space.
  • four Dell R930 nodes, with 2.8Ghz Haswell processors  with 40 cores  and 2,6 TB of  local SSD based fast temporary storage space.

 

Login nodes

In addition to the computing nodes, Taito has four login nodes, two of which (taito-login3 and taito-login4) are used for logging into the system, submitting jobs, I/O and service usage. The tow other login nodes act as the front ends for the GPGPU and MIC hardware linked to the cluster.  The login nodes are HP Proliant DL380 G8.

Interconnect

Communication among nodes and to the storage is done by Infiniband FDR fabric, which provides low latency and high throughput connectivity. High speed interconnect is provided by 58 Mellanox Infiniband FDR switches with 36 ports each, and by the Infiniband HCAs installed on each computing node. The network topology for the cluster is 4:1 pruned tree fabric.

Table 1.2 Configuration of the Taito.csc.fi supercluster. The aggregate performance of the system is 600 TF/s.

Node type Number of nodes Node model Number of cores / node Total number of cores Memory / node

Sandy Bridge login node

4 HP Proliant DL380 G8 16 64 64 / 192 GB
Haswell compute node 397 HP Apollo 6000 XL230a G9 24 9528 128 GB
Haswell big memory node 10 HP Apollo 6000 XL230a G9 24 240 256 GB
Sandy Bridge big memory node (these nodes has been removed) 16 HP SL230s G8 16 256 256 GB
Sandy Bridge huge memory node 2 HP Proliant DL560  32 64 1,5 TB
Haswell huge memory node 4 Dell R930 40 160 1,5 TB

The following commands can give some useful information from the whole Taito system or from the current node a user is logged in.

To get a quick overview of all Taito compute node status use the following command:

sinfo -Nel

The above command prints information in a compute node oriented format. Alternatively, you can get the information in in a partition/queue oriented format with command:

sinfo -el 

For information about the disk systems one can use the following command:

df -h

Details about the available processors on the current node can be checked with:

cat /proc/cpuinfo

And details about the current memory usage on the node is shown with:

cat /proc/meminfo

 

    Previous chapter     One level up     Next chapter