Data Transfer with GridFTP

This page describes how to set up a Globus GridFTP client on your linux system to perform high data rate transfers to retrieve large amounts of data from HPC systems. GridFTP is available for our systems HERMIT and LAKI.

Introduction

For transfering large amounts of data, simple FTP protocol can not utilize high bandwidth channels. For this task, an extension has been definied: GridFTP supports parallel TCP streams and multi-node transfers to achieve a high data rate via high bandwidth connections. Furthermore, transfers can be restarted and third-party transfers can be established. This means one can initiate transfers between two end hosts that are mediated by a third party.

GridFTP has a typical client/server architecture, where the server stores the data or has access to the data. A simple GridFTP client - globus-url-copy - is provided by the Globus Toolkit.

Requirements

• A personal X509 certificate. (Further information: http://en.wikipedia.org/wiki/X.509 , http://www.eugridpma.org/members/worldmap/ , http://www.prace-project.eu/Certificates-FAQ?lang=en)
• The DN of this certificate has to be extracted and sent to us. One can extract the DN by the following command:

openssl x509 -subject -in <USERCERT> -noout | sed -e 's/subject= //'

• A Linux System with the GridFTP client installed

Installation & Configuration

• Since the version 5.2, the GridFTP client is also available packaged as rpm- or deb-package. Install the GridFTP client by following the instructions on this page Be sure to have "globus-proxy-utils" as well (If it the client ist compiled from source, this is included).

• Create a directory .globus/ in your homedir and place both the certificate and your keyfile into this directory

• In the above directory, create another directory certificates/ and place all the CA files there. These files can be found at e.g. here as a tarball- just untar them into the above directory. These files are needed to later verify your certificate against the Certificate Authority.

• run

 grid-proxy-init 

This tool verifies the validity of your certificate and creates a proxy, that is internally needed by the GridFTP client. This step has to be repeated before the usage. If something like

Your identity: <YourDNhere>
Creating proxy ............................................... Done
Your proxy is valid until: Wed Apr 18 22:25:32 2012

shows up, everything is installed correctly.

Firewall issues

Because there is a distinction between control and data connection, some ports of the firewall on the client side have to be opened:

• Port 2812 for the control channel to the frontend node gridftp-fr1.hww.de (HERMIT) or gridftp-fr2.hww.de (LAKI)
• Ports 20000-20500 for data channels to the backend node (Hostnames on request. These ports have to be opened for both incoming and outgoing connections.

Moreover, you have to set the following environment variables to instruct your client to use the specified ports:

export GLOBUS_TCP_PORT_RANGE=20000,20500
export GLOBUS_TCP_SOURCE_RANGE=20000,20500

Please refer to this document for further information: http://www.globus.org/toolkit/docs/latest-stable/gridftp/user/#gridftp-user-config-client-firewall

Usage

See http://www.globus.org/toolkit/docs/5.0/5.0.0/data/gridftp/pi/#globus-url-copy for details of the globus-url-copy tool

The basic syntax is:

globus-url-copy [optional command line switches] source destination

where source and destination can be further resolved to

globus-url-copy [optional command line switches] [gsiftp://<server adress>:<port> | file://]<absolute path> [gsiftp://<server adress>:<port> | file://]<absolute path>

Files on remote systems are referenced by gsiftp:// whereas local files a referenced by file://. Be sure always to reference the absolute paths.

To access files on HERMIT, the informations are:

• server adress: gridftp-fr1.hww.de
  
• port: 2812
  

To access files on LAKI, the informations are:

• server adress: gridftp-fr2.hww.de
  
• port: 2812
  

For the referenced directories, you have to specify the absolute path to your workspace. If you are logged into HERMIT, you can find out about your workspace with the command

ws_list

that lists all your available workspaces. Your workspace will reside in a directory like

/univ_1/ws1/ws/<username-name>

Suppose your workspace directory is

/univ_1/ws1/ws/foo-test-0

and you want to copy files from this workspace to the home directory of the machine you are currently logged in, perform these commands:

grid-proxy-init

globus-url-copy -tcp-bs 4000000 -p 8 gsiftp://gridftp-fr1.hww.de:2812/univ_1/ws1/ws/foo-test-0/file  file:///home/foo/file

It may be neccessary to play around with the parameters a little bit to achieve optimal performance

Some important parameters

Parameter	Description
-help	Prints out a detailled list of parameters and their description
-vb	Verbose mode, show more information: number of bytes transferred, performance since the last update (every 5 seconds) and average performance for the whole transfer
-dbg	Debug mode, gives detailed information for debugging
-p	Specifies the number of the parallel streams.
-tcp-bs	Specifies the size (in bytes) of the TCP buffer to be used by the underlying ftp data channels. Please note that while higher values yield better performance, many parallel streams (high p) together with large buffer sizes could drive the systems out of memory.

Further Information

• http://www.globus.org/toolkit/docs/latest-stable/gridftp/ Offical documentation
• http://www.prace-ri.eu/Data-Transfer-with-GridFTP-Details Intend to PRACE user, but could also be helpful to others
• http://www.globus.org/toolkit/docs/latest-stable/gridftp/user/#gridftp-user-config-client-firewall Firewall issues

Support

Björn Schembera schembera@hlrs.de