Data Transfer with UFTP

Currently in testing, not production ready

Introduction

UFTP is a data streaming library and file transfer tool. It is integrated into UNICORE, allowing to transfer data from client to server (and vice versa), as well as providing data staging between UFTP-enabled UNICORE sites. UFTP can also be used independently from UNICORE, requiring a authentication server and a standalone UFTP client.

UFTP can also be used independently from UNICORE, requiring an authentication server and a standalone UFTP client. You can install the standalone client in your desktop and use it for copying data from or to the file systems of the Supercomputers

Features

• dynamic firewall port opening using a pseudo FTP connection. UFTPD requires only a single open port.
• optional encryption of the data streams using a symmetric key algorithm
• optional compression of the data streams (using gzip)
• partial reads/writes to a file. If supported by the filesystem, multiple UFTP processes can thus read/write a file in parallel (striping)
• supports efficient synchronization of single local and remote files using the rsync algorithm
• integrated into UNICORE clients for fast file upload and download
• integrated with UNICORE servers for fast data staging and server-to-server file transfers
• standalone (non-UNICORE) client available

How does UFTP works?

The server part, called uftpd, listens on two ports (which may be on two different network interfaces):

• the command port receives control commands (for connections from authentication server)
• the listen port accepts data connections from clients.

The uftpd server is "controlled" (usually by UNICORE/X) via the command port, and receives/sends data directly from/to a user’s client machine or another UFTP enabled UNICORE server. Data connnections are made to the "listen" port, which has to be accessible from external machines. Firewalls have to treat the "listen" port as an FTP port. A UFTP file transfer works as follows:

• the UNICORE/X server (or authentication server) sends a request to the command port. This request notifies the UFTPD server about the upcoming transfer and contains the following information
  • the client’s IP address
  • the source/target file name
  • whether to send or receive data
  • a "secret", i.e. a string the client will send to authenticate itself
  • how many data connections will be opened
  • the user and group id for who to create the file (in case of send mode)
  • an optional key to encrypt/decrypt the data
• the UFTPD server will now accept an incoming connection from the announced IP address, provided the supplied "secret" matches the expectation.
• if everything is OK, the requested number of data connections from the client can be opened. Firewall transversal will be negotiated using a pseudo FTP protocol.
• the file is sent/received using the requested number of data connections
• to access the requested file, uftpd attempts to switch its user id to the requested one prior to reading/writing the file. This uses a C library which is accessed from Java via the Java native interface (JNI). See also the installation section below.

Setup at HLRS

For using Unicore UFTP, you need to have console access to a system where either the source or destination filesystem is directly mounted/accessible. The other side needs to have a running UFTPD server with access to the desination/source filesystem mounted.

Here at HLRS we have 5 data transfer nodes (data transfer backends/BE) available and two authentication server (data transfer frontends/FE). The main systen is gridftp-fr1 using the majority of BEs for transfers. gridftp-fr2 is for testing purpose and the available bandwidth is limited.

Possible scenarios (unfinished)

• Upload/download from/to a your home site (normally not running UFTP servers) to HLRS
• Upload/download from a UFTP enabled site (like GCS Centers) from/to HLRS

You have then two options

• • Initiating the transfer from the remote site
  • Initiating the transfer from HLRS

UFTP client (unfinished)

The UFTP standalone Client is a Java-based client for UFTP and runs under Linux. It allows to list remote directories, copy files (with many options such as wildcards), and sync single files. It supports username/password authentication and ssh-key authentication to a UFTP Authentication Server.

Installation (unfinished)

The Unicore download page (https://www.unicore.eu/download/) contains all components for Unicore. Unless you want to use the full Unicore potential, the UFTP Client is enough and you can use the direct link to sourceforge (http://sourceforge.net/projects/unicore/files/Clients/UFTP-Client/). The directory for a version contains three installation method

• RPM based version (for Redhat based systems, also CentOS, Scientific Linux, ...)
• DEB based version (for Debian based systems, also Ubuntu, ...)
• tar.gz version (for all systems including above mentioned ones)

You can use any of the listed installation methods. Since RPM and DEB should be clear, we explain the manual installation.