- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

HPSS User Access: Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
m (User Access moved to HPSS User Access: misleading name)
m (How to close ftp)
 
(20 intermediate revisions by 6 users not shown)
Line 1: Line 1:
Currently, there are two methods to access the HPSS complex:
= General =


== FTP ==
Please be aware that HPSS is a Hierarchical Storage Management System (HSM) with tape storage included; certain characteristics differ from a disk usage.
Initiate the ftp connection to HPSS using
When transferring a file to the system via (p)ftp it is stored on the disk cache as first step. Later, the data is migrated to two copies on tape.
When retrieving a file, it may happen that the file is not on the disk cache anymore and has to be recalled from tape storage. This can take a few minutes.


  ftp hpsscore 4021
= Best Practices =


when requested, please provide your username and password.
* Do not store large numbers of files.
The password is your general hww password (as your have it
* It is not recommended to recursively store directory structures.
for the SX-8 for example).
* Better tar small files to one larger archive. Tar can also be used to keep the directory structures.
* HPSS is not meant as a backup system.
* Pftp is preferred since it automatically choses the right Class of Service (CoS) and is much faster due to parallel streaming of data.


'''Please note:''' For technical reasons a password change
= Access Methods =
is vald on HPSS with a one day delay.


=== Class of Service ===
== Parallel FTP ==
When putting files into HPSS you have to specify a so called
class of service. The class of service depends on the file size.
The following values are possible:


112: size < 2GB
On Vulcan and HAWK there is a Parallel FTP client available.


122: 2GB < size < 8GB
The advantage of Parallel FTP is a file transfer which makes use of several parallel
network connections or at least several parallel I/O streams.


132: size > 8GB
On Vulcan and HAWK, you have to load the module "hpss" first:  


You have to specify the class of service for a file prior to
{{Command| command=module load hpss}}
moving the file into HPSS. Before the transfer, you have to specify at the
ftp prompt:
  site setcos <cos id>


Default setting for the cos id is 112. The setting of the cos id is valid for all
To start the client, you should specify the number of parallel streams. We recommend a setting of 4.
following file transfers.


Beside the setting of the class of service, this option works just like normal ftp. 
{{Command| command=pftp_client -w 4 hpsscore 4021}}


When requested, please provide your username and password.
The password is your general hww password (as you have it
for HAWK for example).


{{Note| text=For technical reasons a password change
is valid on HPSS with a one day delay.}}


== Parallel FTP ==
Compared to the ftp access, there is no setting for the class of service necessary as this is done automatically by the pftp_client.


on the following systems, there is a Parallel FTP client available:
Use the parallel ''pput'' and ''pget'' commands instead of the normal ''put'' and ''get''.


asama.hww.de, a1.hww.de
In this example, the user hpctest uses the pput command to copy an archive from his/her workspace to the /backup directory on HPSS.


The advantage of Parallel FTP is a file transfer which makes use of several parallel
{{Command
network connections or at least several parallel I/O streams.
| command =
$ cd /lustre/hpe/ws<number>/<Project> <br />


to start the client, you should specify the number of parallel streams.
$ module load hpss <br />
$ pftp_client -w 4 hpsscore.hww.de 4021 <br />
220 hpsscore.hww.de FTP server (HPSS 7.5.1 Parallel FTP Daemon) ready.<br />
Parallel stripe width set to (4).<br />
Name (hpsscore.hww.de:hpctest): hpctest<br />
331 Password required for hpctest.<br />
Password:***************<br />
230 User hpctest logged in as hpctest@HPSSCORE.HWW.DE<br />
Remote system type is UNIX.<br />
215 Parallel protocol is PDATA_AND_MOVER<br />
215 SAN3P support disabled<br />


We recommend a setting of 2.
ftp> mkdir backup<br />
ftp> cd backup<br />
ftp> pput archive.tar.gz<br />
ftp> quit
}}


  pftp_client -w2 hpsscore 4021
More details of using ftp and pftp to access HPSS can be found in
the official [http://www.hpss-collaboration.org/user_doc.shtml User's guide].
 
== FTP ==
Initiate the ftp connection to HPSS using
 
{{Command| command=ftp hpsscore.hww.de 4021}}


When requested, please provide your username and password.
When requested, please provide your username and password.
The password is your general hww password (as your have it
The password is your general hww password (as you have it
for the SX-8 for example).
for Hermit for example).
 
ftp will check the <tt>.netrc</tt> file in the user's home directory,
where e.g. login information can be given and the the ftp command
can be used in e.g. scripts without a password prompt (see ftp man pages).
Be aware of security issues, if a password is stored in plain text
in a .netrc file.
 
{{Note| text=For technical reasons a password change
is valid on HPSS with a one day delay.}}
 
=== Class of Service ===
When putting files into HPSS you have to specify a so called
class of service. The class of service depends on the file size.
The following values are possible:
 
102: size < 500MB
 
122: 0.5GB < size < 8GB


'''Please note:''' For technical reasons a password change
132: size > 8GB
is valid on HPSS with a one day delay.
 
You have to specify the class of service for a file prior to
moving the file into HPSS. Before the transfer, you have to specify at the
ftp prompt:
{{Command| command=site setcos <cos id>}}
The setting of the cos id is valid for all
following file transfers.


Compared to the ftp access, there is no setting for the class of service
Beside the setting of the class of service, this option works just like normal ftp.
necessary as this is done automatically by the pftp_client.


For a parallel put please use the pput command instead of put.
For a parallel get, please use pget.


More details of using ftp and pftp to access HPSS can be found in
the official user guide which will be available here soon.


== Other Access Methods ==
== Other Access Methods ==
Line 71: Line 112:
there is currently no access to HPSS available from outside the hww
there is currently no access to HPSS available from outside the hww
network. For a transfer of stored files to your home site,
network. For a transfer of stored files to your home site,
please move these files to a system within hww where you have scp access.
please move these files to a system within hww where you have scp or GridFTP access.
This is an intermediate solution, we plan to provide a method for a
 
direct transfer within the next months.
= Further Information =
 
The official user guide by IBM can be found here: http://www.hpss-collaboration.org/documents/hpss752/Users_Guide.pdf

Latest revision as of 09:34, 21 March 2024

General

Please be aware that HPSS is a Hierarchical Storage Management System (HSM) with tape storage included; certain characteristics differ from a disk usage. When transferring a file to the system via (p)ftp it is stored on the disk cache as first step. Later, the data is migrated to two copies on tape. When retrieving a file, it may happen that the file is not on the disk cache anymore and has to be recalled from tape storage. This can take a few minutes.

Best Practices

  • Do not store large numbers of files.
  • It is not recommended to recursively store directory structures.
  • Better tar small files to one larger archive. Tar can also be used to keep the directory structures.
  • HPSS is not meant as a backup system.
  • Pftp is preferred since it automatically choses the right Class of Service (CoS) and is much faster due to parallel streaming of data.

Access Methods

Parallel FTP

On Vulcan and HAWK there is a Parallel FTP client available.

The advantage of Parallel FTP is a file transfer which makes use of several parallel network connections or at least several parallel I/O streams.

On Vulcan and HAWK, you have to load the module "hpss" first:

module load hpss


To start the client, you should specify the number of parallel streams. We recommend a setting of 4.

pftp_client -w 4 hpsscore 4021


When requested, please provide your username and password. The password is your general hww password (as you have it for HAWK for example).

Note: For technical reasons a password change is valid on HPSS with a one day delay.


Compared to the ftp access, there is no setting for the class of service necessary as this is done automatically by the pftp_client.

Use the parallel pput and pget commands instead of the normal put and get.

In this example, the user hpctest uses the pput command to copy an archive from his/her workspace to the /backup directory on HPSS.

$ cd /lustre/hpe/ws<number>/<Project>

$ module load hpss
$ pftp_client -w 4 hpsscore.hww.de 4021
220 hpsscore.hww.de FTP server (HPSS 7.5.1 Parallel FTP Daemon) ready.
Parallel stripe width set to (4).
Name (hpsscore.hww.de:hpctest): hpctest
331 Password required for hpctest.
Password:***************
230 User hpctest logged in as hpctest@HPSSCORE.HWW.DE
Remote system type is UNIX.
215 Parallel protocol is PDATA_AND_MOVER
215 SAN3P support disabled

ftp> mkdir backup
ftp> cd backup
ftp> pput archive.tar.gz

ftp> quit


More details of using ftp and pftp to access HPSS can be found in the official User's guide.

FTP

Initiate the ftp connection to HPSS using

ftp hpsscore.hww.de 4021


When requested, please provide your username and password. The password is your general hww password (as you have it for Hermit for example).

ftp will check the .netrc file in the user's home directory, where e.g. login information can be given and the the ftp command can be used in e.g. scripts without a password prompt (see ftp man pages). Be aware of security issues, if a password is stored in plain text in a .netrc file.

Note: For technical reasons a password change is valid on HPSS with a one day delay.


Class of Service

When putting files into HPSS you have to specify a so called class of service. The class of service depends on the file size. The following values are possible:

102: size < 500MB

122: 0.5GB < size < 8GB

132: size > 8GB

You have to specify the class of service for a file prior to moving the file into HPSS. Before the transfer, you have to specify at the ftp prompt:

site setcos <cos id>

The setting of the cos id is valid for all following file transfers.

Beside the setting of the class of service, this option works just like normal ftp.


Other Access Methods

Currently, there are no other access methods available. Especially, there is currently no access to HPSS available from outside the hww network. For a transfer of stored files to your home site, please move these files to a system within hww where you have scp or GridFTP access.

Further Information

The official user guide by IBM can be found here: http://www.hpss-collaboration.org/documents/hpss752/Users_Guide.pdf

Retrieved from "https://kb.hlrs.de/platforms/index.php?title=HPSS_User_Access&oldid=7309"