Gis Data in Taito

There exists a shared data folder in taito at /proj/ogiir-csc where some datasets from for example MML, LUKE and FMI can be found. The obvious advantage of using this shared data is that especially for large datasets such as lidar data, it isn't necessary to waste time for moving data back and forth between Taito, data provider and end user.

Currently (July 2017) there are following datasets:

  • MML:
    • Lidar point cloud data
    • Dem 2m (see virtual rasters section below)
    • Dem 10m (see virtual rasters section below)
  • FMI
    • 10km avg relative humidity
    • 10km avg sea level preassure
    • 10km daily max temperature
    • 10km daily mean temperature
    • 10km daily min temperature
    • 10km daily precipitation
    • 10km daily radiation
    • 10km daily snow
    • 10km monthly mean temperature
    • 10km monthly precipitation
  • LUKE
    • Multi-source national forest inventory

An up to date situation can be easily checked from taito by browsing /proj/ogiir-csc folder.

Readymade virtual rasters in Taito

There is a ready made structure of virtual rasters for 2m and 10m elevation model in /proj/ogiir-csc/mml/dem2m_vrt and /proj/ogiir-csc/mml/dem10m_vrt folders. There are two variants of virtual rasters for these elevation models: 

  1. The hierarchical virtual raster is mainly for viewing purposes for example with QGIS. It has a hierarchical structure where a virtual raster for each folder contains all the data stored in that folder and it's subfolders. For example if I wanted to view 2m dem from area of mapsheet M4 dem I would simply open the dem2m_vrt/etrs-tm35fin-n2000/M4/M4.vrt file which would then load virtual rasters for mapsheets M41, M42, M43 and M44 which in turn contain information about the actual tif files. If i wanted to view the whole 2m dem dataset I would simply open dem2m_vrt/etrs-tm35fin-n2000/whole_finland.vrt.

    The hierarchical file structure also contains statistics (min, max, mean, stddev) and overviews for each vrt file which enables a fairly responsive viewing of entire 2m or 10m dem for example in QGIS. A good tip is to enable the Raster toolbar through View->Toolbars which allows to easily adjust the color scale to min-max of current view extent. This way the whole dataset can be easily viewed at different zoom levels.

    You may use the lowest level virtual raster (for example M41 in the 2m DEM) also in scripts, higher level virtaul rasters may cause computational errors.
     
  2. The direct virtual rasters contain directly the source tif images without any hierarchical structure, overviews or pre-calculated statistics. The direct virtual raster is meant for using only in scripts (it should not be opened in QGIS etc).

The optimal performance for analysis will always be achieved by creating your own virtual raster that covers only your study area.

 

Creating virtual rasters for specific area

There is a ready made python script for creating virtual rasters for your study area from datasets available in Taito. The script is located at /proj/ogiir-csc/mml/karttalehtijako/vrt_creator.py. It's used in a following way:

python /proj/ogiir-csc/mml/karttalehtijako/vrt_creator.py dataset polygon_file output_directory

available dataset values are:

  • dem2m - 2m dem from mml located at /proj/ogiir-csc/mml/dem2m
  • dem10m - 10m dem from mml located at /proj/ogiir-csc/mml/dem10m
  • demCombined - dem covering whole Finland using 2m dem whenever it's available and interpolating rest of the areas to 2m resolution from 10m dem using bicubic interpolation.

with optional arguments:

  • -i: create individual vrt for each polygon, default behavior is to create one vrt covering all polygons.
  • -o: create overviews
  • -p:output name prefix

The script utilizes gdal and geopandas python module so geo-env module should be loaded prior to using the script. For more details see: https://research.csc.fi/-/geopython

Example: create one virtual raster covering my_study_area.shp polygon file from 2m dem found at /proj/ogiir-csc/mml/dem2m and save it to /wrk/USERNAME/output/2m_dem_virtual_raster.vrt. A list of files will also be saved to /wrk/USERNAME/output/2m_dem_file_list.txt as well as external overview file /wrk/USERNAME/output/2m_dem_virtual_raster.vrt.ovr

python /proj/ogiir-csc/mml/karttalehtijako/vrt_creator.py dem2m my_study_area.shp /wrk/USERNAME/output -o -p 2m_dem