Taito and HPC Archive are about to be closed
Taito will be generally available to users until 31.12.2019. This means that it is high time to make sure that your workflows run on Puhti, and to move all relevant data to Allas (or elsewhere) from home, work and project directories. To ease the transition to Mahti, CSC will operate Taito with limited support and low service level in early 2020. In practice it means that
1) Support and user documentation is not available for Taito in 2020 (support for data migration is provided)
2) New features are not implemented, and new software is not installed
3) Taito, or parts of Taito, may be permanently closed at any time with no warning. Taito will be fully decommissioned at the latest in March 2020.
HPC archive service will also be shut down in 2020. For the archive, you have time until the end of 2020 to move data, and it does not need to be migrated right now. No new data should be written to archive anymore. In January 2020 the archive will be set in read-only mode. For larger datasets (more than 1 TiB) we kindly ask you to contact CSC servicedesk so that the datatransfers can be done efficiently. Further details on transferring data from HPC Archive will be provided later.
Migrating data from Taito
Make sure to move all data in 2019. After 31.12.2019 CSC cannot guarantee access to the data. The datamangler nodes and Taito's $WRK, $PROJ and $HOME folders will be available also after new year for a limited amount of time, but in case of critical software or hardware problems the file system may be permanently closed with no warning and all data on the system may be lost.
We recommend that the data is transferred using the datamangler.csc.fi nodes that give you access to Taito, Allas and the Fairdata services (IDA). Datamangler is new temporary service to support data migration from Taito to Allas. The transfer capacity of Datamangler is significantly greater than from Taito login nodes or Taito-shell. All of them have the same data transport commands and procedures available.
In general we recommend to move the data to Allas. The new Allas object storage service provides a platform to store your data that is currently in the disks of Taito. HPC-archive should not be used store any new data.
Practical aspects
1. Get access to Allas
By default the CSC computing projects do not have access to Allas. Thus, the first thing to do is to add Allas service for your project. This is done with the MyCSC interface. (instructions)
Please note that only the project manager can apply for the access. Once access is granted, all project members must visit MyCSC service and accept the terms of use for Allas before they can use the Allas storage area.
2. Make sure you have enough quota to store your data
The default storage quota in Allas is 10 TB. As this space is shared with all project members, it is quite possible that this is not enough. In that case you should estimate how much space is needed and then send a request for more space. Note that files stored in Allas consume billing units, 1 BU per TiB hour. The request should be sent to servicedesk@csc.fi. Please include to your quota request:
- ID/name of your project
- amount of Allas space needed
- a short description of the data to be stored
3. Transfer using datamangler nodes
Datamangler should be used for data transport only. To access Datamangler, use address: datamangler.csc.fi. For example
ssh csc-user-account@datamangler.csc.fi
When planning the transfer keep in mind that:
1. Delete data from Taito once you have verified the data has been successfully transferred to Allas. If you have data that has no value then please delete it immediately. This enables us to better monitor the amount of data that still needs to be migrated off Taito.
2. Try to import data to Allas in chunks that are smaller than 1 TB
3. Moving tens of Terabytes of data can take several days.
4. a-put command uses data compression by default when storing data to Allas. This is good approach for ASCII formatted data (i.e. data consisting of letters and numbers). If your data is in binary format that does not compress much, it is faster to use rclone command for the data upload.
5. At the moment concentrate to the data in the disk areas of Taito. HPC-archive will be available for a bit longer time than Taito.
6. You can have several data upload processes running in Datamangler at the same time.
7. Please, contact servicedesk@csc.fi if you have any questions related to moving your data away from Taito.
Further information
You can find Allas user guide here: https://docs.csc.fi/data/Allas/
And datamigration examples here: https://docs.csc.fi/data/Allas/migration_tutorial/