- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

Workspace migration: Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
No edit summary
 
(10 intermediate revisions by one other user not shown)
Line 2: Line 2:


{{Warning
{{Warning
| text = This page describe the necessary steps to migrate workspaces to the another workspace filesystems.<br/>  
| text = This page describes the necessary steps to migrate workspaces to another workspace filesystems.<br/>  


On 18th October of 2024 all data on the old hawk ws10 filesystems will be deleted. Follow the below guide to transfer your data to the ws11 workspace filesystems.
On 18th October 2024 all data on the old Hawk ws10 filesystems will be deleted. Follow the guide below to transfer your data to the ws11 workspace filesystems.


}}
}}
Line 12: Line 12:




On hawk the workspace filesystem ws11 will be the new default workspace filesystem. The currently existing default workspace filesystem (ws10) will be shut down on 18th October 2024. The policy settings from ws10 will be transferred to the ws11 filesystem.
On Hawk the workspace filesystem ws11 will be the new default workspace filesystem. The currently existing default workspace filesystem (ws10) will be shut down on 18th October 2024. The policy settings of ws10 will be transferred to the ws11 filesystem.
Now, users have to migrate their workspaces located on the ws10 filesystems onto the ws11 filesystem. Run the command ''ws_list -a'' on a frontend system to display the path for all your workspaces, if path names match mount points in the following table, these workspaces need to  
Now, users have to migrate their workspaces located on the ws10 filesystems onto the ws11 filesystem. Run the command ''ws_list -a'' on a frontend system to display the path for all your workspaces. If path names match mount points listed in the following table, these workspaces need to  
be migrated to the ws11 filesystem.
be migrated to the ws11 filesystem.


Line 43: Line 43:
Migration for large amount of data consumes a lot of IO ressources. '''Please review and remove data not needed any more or move it into [[High_Performance_Storage_System_(HPSS)| HPSS]].'''
Migration for large amount of data consumes a lot of IO ressources. '''Please review and remove data not needed any more or move it into [[High_Performance_Storage_System_(HPSS)| HPSS]].'''


== How to proceed / time schedule for the gradual shutdown of ws10 ==
== How to proceed / Time schedule for the gradual shutdown of ws10 ==


* <Font color=red>2024-07-22 10:00</Font>:
* <Font color=red>2024-07-22 10:00</Font>:
** the default workspace filesystem will be switched from ws10 to ws11
** The default workspace filesystem will be switched from ws10 to ws11. ws_* tools applied to ws10 require the additional option <tt>'-F <workspacefilesystem>'</tt>.
** ws11 will get the [[Workspace_migration#Operation_of_the_workspaces_on_ws11: |same policy settings]] as ws10
** ws11 will get the [[Workspace_migration#Operation_/_Policies_of_the_workspaces_on_ws11: |same policy settings]] as ws10.
** <Font color=red>Quota limit settings for each group will be moved from ws10 to ws11 and the quota enforcement will be enabled on ws11! </Font>
** <Font color=red>Quota limit settings for each user-group will be moved from ws10 to ws11 and the quota enforcement will be enabled on ws11! (Batchjobs will not be scheduled in case the quota was exceeded the limit for your group)</Font>.
** reducing the max extension for workspaces on ws10 from 3 to 1 (ws_allocate, ws_extend)  
** Max. extension for workspaces on ws10 will be reduced from 3 to 1 (ws_allocate, ws_extend).
** reducing the max duration for workspaces on ws10 from 60 days to 40 days (ws_allocate, ws_extend)
** Max. duration for workspaces on ws10 will be reduced from 60 days to 40 days (ws_allocate, ws_extend).


* <Font color=red>2024-08-19</Font>:
* <Font color=red>2024-08-19</Font>:
** reducing the max durations for workspaces on ws10 from 40 days to 10 days (ws_allocate)
** Max. durations for workspaces on ws10 will be reduced from 40 days to 10 days (ws_allocate).
** deactivation of ws_extend for ws10
** Deactivation of ws_extend for ws10.


* <Font color=red>2024-09-09</Font>:
* <Font color=red>2024-09-09</Font>:
** reducing the max duration for workspaces on ws10 from 10 days to 3 days (ws_allocate)
** Max. duration for workspaces on ws10 will be reduced from 10 days to 3 days (ws_allocate).


* <Font color=red>2024-10-18</Font>:
* <Font color=red>2024-10-18</Font>:
** start removing all remaining data located on ws10
** Start removing all remaining data located on ws10.
** final shutdown
** Final shutdown.


* from <Font color=red>2024-07-22 10:00</Font>: switching default workspace filesystem from ws10 to ws11. ws11 has the [[Workspace_migration#Operation_of_the_workspaces_on_ws11: |same policy settings]] as ws10.
=== Important remarks ===
** From that point new workspaces will be allocated on the ws11 filesystem.
* If you have to migrate data residing in workspaces from one to another filesystem, do not use the ''mv'' command to transfer data. For large amounts of data, this will fail due to time limits. Currently we recommend for e.g. millions of small files or for large amounts of data to use the following command inside a single node batch job: ''rsync -a  --hard-links  Old_ws/  new_ws/'' .
** Existing workspaces on ws10 will be listed further on.
* Please try to use the [[Workspace_migration#Using_mpifileutils_for_data_transfer | mpifileutils '''dcp''' or '''dsync''']].
** Creating new workspace on ws10 is disabled.
* Take care when you create new batch jobs. If you have to migrate your workspace from an old filesystem to the new location, this takes time. Do not run any job while the migration process is active. This may result in inconsistent data.
** ws_* tools applied for ws10 additional requires the option <tt>'-F <workspacefilesystem>'</tt>
** Quota limit settings for ws11 will be the same as on ws10.
* from <Font color=red>August 15th 2024 10:00</Font>:
** Restoring expired workspaces on ws10 is disabled.
** workspaces located on ws10 filesystems can not be extended anymore.
* if you have to migrate data from workspaces on one to another filesystems, do not use the ''mv'' command to transfer data. For large amount of data, this will fail due to time limits. Currently we recommend for e.g. millions of small files or for large amount of data to use following command inside a single node batch job: ''rsync -a  --hard-links  Old_ws/  new_ws/''  
* You can try to use the [[Workspace_migration#Using_mpifileutils_for_data_transfer | mpifileutils '''dcp''' or '''dsync''']]
* take care when you create new batch jobs. If you have to migrate your workspace from an old filesystem to the new location, this takes time. Do not run any job while the migration process is active. This may result in inconsistent data.
* On <Font color=red>October 15th 2024</Font> the “old” workspaces on ws10 will be disconnected from the hawk compute nodes. The filesystems will be available on the frontend systems for some further days.
* <Font color=red>November 2024</font> all data on the old ws10 filesystems will be deleted.


== Operation / Policies of the workspaces on ws11: ==
== Operation / Policies of the workspaces on ws11: ==


* No job of any group member will be scheduled for computation as long as the group quota is exceeded.
* No job of any user-group member will be scheduled for computation as long as the group quota is exceeded.
* accounting
* Accounting.
* max. lifetime of a workspace is currently 60 days
* Max. lifetime of a workspace is currently 60 days.
* default lifetime of a workspace is 1 day
* Default lifetime of a workspace is 1 day.
* max. number of workspace extensions is 3
* Max. number of workspace extensions are 3.
* please read related man pages or online [[Workspace_mechanism | workspace mechanism document]]<BR>
* Please read related man pages or the online [[Workspace_mechanism | workspace mechanism document]]<BR>
: in particular note that the workspace tools allow to explicitly address a specific workspace file system using the <tt>-F</tt> option (e.g. <tt>ws_allocate -F ws11.0 my_workspace 10</tt>)
: in particular note that the workspace tools allow to explicitly address a specific workspace file system using the <tt>-F</tt> option (e.g. <tt>ws_allocate -F ws11.0 my_workspace 10</tt>)
* to list your available workspace file systems use <tt>ws_list -l</tt>  
* To list your available workspace file systems use <tt>ws_list -l</tt>.
* users can restore expired workspaces using ''ws_restore''
* Users can restore expired workspaces using ''ws_restore''.


Please read https://kb.hlrs.de/platforms/index.php/Storage_usage_policy
Please read https://kb.hlrs.de/platforms/index.php/Storage_usage_policy


== Using mpifileutils for data transfer ==
== Using mpifileutils for data transfer ==
This mpifileutils suite provides MPI-based tools to handle typical jobs like copy, remove, and compare for large  datasets, providing speedups of up to 50x compared to single process jobs. It can only be run on compute nodes via mpirun.
The mpifileutils suite provides MPI-based tools to handle typical jobs like copy, remove, and compare for large  datasets, providing speedups of up to 50x compared to single process jobs. It can only be run on compute nodes via mpirun.


dcp or dsync is similar to cp -r or rsync; simply give it a source directory and destination and dcp / dsync will recursively copy the source directory to the destination in parallel.
dcp or dsync is similar to cp -r or rsync; simply provide a source directory and destination and dcp / dsync will recursively copy the source directory to the destination in parallel.


dcp / dsync has a number of useful options; use dcp -h or dsync -h to see a description or use the [[https://mpifileutils.readthedocs.io/en/v0.11.1/ User Guide]]
dcp / dsync has a number of useful options; use dcp -h or dsync -h to see a description or use the [[https://mpifileutils.readthedocs.io/en/v0.11.1/ User Guide]].


It should be invoked via mpirun.
It should be invoked via mpirun.

Latest revision as of 07:23, 15 July 2024


Warning: This page describes the necessary steps to migrate workspaces to another workspace filesystems.
On 18th October 2024 all data on the old Hawk ws10 filesystems will be deleted. Follow the guide below to transfer your data to the ws11 workspace filesystems.


User migration to new workspaces

On Hawk the workspace filesystem ws11 will be the new default workspace filesystem. The currently existing default workspace filesystem (ws10) will be shut down on 18th October 2024. The policy settings of ws10 will be transferred to the ws11 filesystem. Now, users have to migrate their workspaces located on the ws10 filesystems onto the ws11 filesystem. Run the command ws_list -a on a frontend system to display the path for all your workspaces. If path names match mount points listed in the following table, these workspaces need to be migrated to the ws11 filesystem.


File System mounted on
ws10.0 /lustre/hpe/ws10/ws10.0
ws10.1 /lustre/hpe/ws10/ws10.1
ws10.2 /lustre/hpe/ws10/ws10.2
ws10.3 /lustre/hpe/ws10/ws10.3
ws10.3P /lustre/hpe/ws10/ws10.3P

Before you start

Migration for large amount of data consumes a lot of IO ressources. Please review and remove data not needed any more or move it into HPSS.

How to proceed / Time schedule for the gradual shutdown of ws10

  • 2024-07-22 10:00:
    • The default workspace filesystem will be switched from ws10 to ws11. ws_* tools applied to ws10 require the additional option '-F <workspacefilesystem>'.
    • ws11 will get the same policy settings as ws10.
    • Quota limit settings for each user-group will be moved from ws10 to ws11 and the quota enforcement will be enabled on ws11! (Batchjobs will not be scheduled in case the quota was exceeded the limit for your group).
    • Max. extension for workspaces on ws10 will be reduced from 3 to 1 (ws_allocate, ws_extend).
    • Max. duration for workspaces on ws10 will be reduced from 60 days to 40 days (ws_allocate, ws_extend).
  • 2024-08-19:
    • Max. durations for workspaces on ws10 will be reduced from 40 days to 10 days (ws_allocate).
    • Deactivation of ws_extend for ws10.
  • 2024-09-09:
    • Max. duration for workspaces on ws10 will be reduced from 10 days to 3 days (ws_allocate).
  • 2024-10-18:
    • Start removing all remaining data located on ws10.
    • Final shutdown.

Important remarks

  • If you have to migrate data residing in workspaces from one to another filesystem, do not use the mv command to transfer data. For large amounts of data, this will fail due to time limits. Currently we recommend for e.g. millions of small files or for large amounts of data to use the following command inside a single node batch job: rsync -a --hard-links Old_ws/ new_ws/ .
  • Please try to use the mpifileutils dcp or dsync.
  • Take care when you create new batch jobs. If you have to migrate your workspace from an old filesystem to the new location, this takes time. Do not run any job while the migration process is active. This may result in inconsistent data.

Operation / Policies of the workspaces on ws11:

  • No job of any user-group member will be scheduled for computation as long as the group quota is exceeded.
  • Accounting.
  • Max. lifetime of a workspace is currently 60 days.
  • Default lifetime of a workspace is 1 day.
  • Max. number of workspace extensions are 3.
  • Please read related man pages or the online workspace mechanism document
in particular note that the workspace tools allow to explicitly address a specific workspace file system using the -F option (e.g. ws_allocate -F ws11.0 my_workspace 10)
  • To list your available workspace file systems use ws_list -l.
  • Users can restore expired workspaces using ws_restore.

Please read https://kb.hlrs.de/platforms/index.php/Storage_usage_policy

Using mpifileutils for data transfer

The mpifileutils suite provides MPI-based tools to handle typical jobs like copy, remove, and compare for large datasets, providing speedups of up to 50x compared to single process jobs. It can only be run on compute nodes via mpirun.

dcp or dsync is similar to cp -r or rsync; simply provide a source directory and destination and dcp / dsync will recursively copy the source directory to the destination in parallel.

dcp / dsync has a number of useful options; use dcp -h or dsync -h to see a description or use the [User Guide].

It should be invoked via mpirun.

We highly recommend to use dcp / dsync with an empty ~/.profile and ~/.bashrc only! Furthermore, take care that only the following modules are loaded when using mpifileutils (this can be achieved by logging into the system without modifying the list of modules and loading only the modules openmpi and mpifileutils):
1) system/site_names
2) system/ws/8b99237
3) system/wrappers/1.0
4) hlrs-software-stack/current
5) gcc/10.2.0
6) openmpi/4.1.4
7) mpifileutils/0.11


dcp

Parallel MPI application to recursively copy files and directories.

dcp is a file copy tool in the spirit of cp(1) that evenly distributes the work of scanning the directory tree, and copying file data across a large cluster without any centralized state. It is designed for copying files that are located on a distributed parallel file system, and it splits large file copies across multiple processes.

Run dcp with the -p option to preserve permissions and timestamps, and ownership.

-p  : preserve permissions and timestamps, and ownership

--chunksize C: Copy files larger than C Bytes in C Byte chunks (default ist 4MB)

We highly recommend using the -p option.

dsync

Parallel MPI application to synchronize two files or two directory trees.

dsync makes DEST match SRC, adding missing entries from DEST, and updating existing entries in DEST as necessary so that SRC and DEST have identical content, ownership, timestamps, and permissions.

--chunksize C: Copy files larger than C Bytes in C Byte chunks (default ist 4MB)


Job Script example

Here is an example of a job script.

You have to change the SOURCEDIR and TARGETDIR according to your setup. Also the number of nodes and wallclock time should be adjusted.


#!/bin/bash
#PBS -N parallel-copy
#PBS -l select=2:node_type=rome:mpiprocs=128
#PBS -l walltime=00:20:00

module load openmpi mpifileutils

SOURCEDIR=<YOUR SOURCE DIRECTORY HERE>
TARGETDIR=<YOUR TARGET DIRECTORY HERE>

sleep 5
nodes=$(cat $PBS_NODEFILE | sort -u | wc -l)
let cores=nodes*20

time_start=$(date "+%c  :: %s")
#mpirun -np $cores dcp -p --bufsize 8MB ${SOURCEDIR}/ ${TARGETDIR}/
mpirun -np $cores dsync --bufsize 8MB $SOURCEDIR $TARGETDIR
time_end=$(date "+%c  :: %s")

tt_start=$(echo $time_start | awk {'print $9'})
tt_end=$(echo $time_end | awk {'print $9'})
(( total_time=$tt_end-$tt_start ))
echo "Total runtime in seconds: $total_time"