3 Directories and data storage at CSC
The CSC supercomputing environment allows researchers to analyse and manage large datasets. Even with a normal CSC user account the researchers can work with datasets that contain several terabytes of data. Each CSC user has access to four personal disk areas that can be used for processing data. For long term data storage, customers of CSC can use HPC archive server or IDA service.
Knowing the basic features of different disk storage areas is essential, if you wish to use the CSC computing services effectively.
Table 3.1: Standard user directories at CSC. The HPC Archive is used through iRODS commands, and it is not mounted to CSC computing environment as a directory. Directories that can be accessed via Scientist's User Interface web portal are marked with an asterisk (*).
|Directory or storage area||Intended use||Default quota/user||Storage time||Backup|
|$HOME *||Initialization scripts, source codes, small data files. |
Not for running programs or research data.
|50 GB||Data will be deleted 90 days after closing the account||Yes|
|$USERAPPL||Users' own application software.||50 GB||Data will be deleted 90 days after closing the account||Yes|
|$WRKDIR *||Temporary data storage.||5 TB||90 days||No|
|$TMPDIR||Temporary users' files.||2 days||No|
|project||Common storage for project members. A project can consist of one or more user accounts.||On request.||Data will be deleted 90 days after closing the project||No|
|HPC Archive *||Long term storage.||2,5 TB||Permanent||Two copies maintained|
|IDA||Long term storage.||On request||Permanent||Part of the Open Science and Research services.|
|Pouta Object Stoirage||Storage and sharing||1 TB||Permanent|
The directories listed in the table above can be accessed through linux command line usage. $HOME, $WRKDIR and HPC Archive can also be accessed through the Scientist's User Interface web portal (https://sui.csc.fi). When you are working on command line, you can utilize automatically defined environment variables that contain the directory paths to different disk areas (excluding project disk for which there is no environment variable). So, if you would like to move to your work directory you could do that with command:
similarly, copying a file data.txt to your work directory could be done with command:
cp data.txt $WRKDIR/
In the following chapters you can find more detailed introductions to the usage and features of different user specific disk areas.
3.1.1 Monitoring disk usage
3.1.2 Home directory, $HOME
3.1.3 Work directory, $WRKDIR
3.1.4 Temporary directory, $TMPDIR
3.1.5 Software installation directory $USERAPPL
Example 1. Installing your own version of program MCL program
Example 2. Installing and using your own Perl module to Taito
Example 3. Installing and using your own Python module to Taito
3.2.1 Using HPC archive
Example 1. Storing data to the HPC archive server
Example 2. Retrieving data from the archive server
3.2.2 Configuring the connection to IDA
Example 3. Storing data to the IDA server
3.2.3 Using HPC Archive and Ida with Scientist's User Interface
|Previous chapter||One level up||Next chapter|