Data Transfer with UFTP

Currently in testing, not production ready

Introduction

UFTP is a data streaming library and file transfer tool.

It is integrated into UNICORE, allowing to transfer data from client to server (and vice versa), as well as providing data staging between UFTP-enabled UNICORE sites. UFTP can also be used independently from UNICORE, requiring a authentication server and a standalone UFTP client.

Features

• dynamic firewall port opening using a pseudo FTP connection. UFTPD requires only a single open port.
• optional encryption of the data streams using a symmetric key algorithm
• optional compression of the data streams (using gzip)
• partial reads/writes to a file. If supported by the filesystem, multiple UFTP processes can thus read/write a file in parallel (striping)
• supports efficient synchronization of single local and remote files using the rsync algorithm
• integrated into UNICORE clients for fast file upload and download
• integrated with UNICORE servers for fast data staging and server-to-server file transfers
• standalone (non-UNICORE) client available

How does UFTP works?

The server part, called uftpd, listens on two ports (which may be on two different network interfaces):

• the command port receives control commands (for connections from authentication server)
• the listen port accepts data connections from clients.

The uftpd server is "controlled" (usually by UNICORE/X) via the command port, and receives/sends data directly from/to a user’s client machine or another UFTP enabled UNICORE server. Data connnections are made to the "listen" port, which has to be accessible from external machines. Firewalls have to treat the "listen" port as an FTP port. A UFTP file transfer works as follows:

• the UNICORE/X server sends a request to the command port. This request notifies the UFTPD server about the upcoming transfer and contains the following information
  • the client’s IP address
  • the source/target file name
  • whether to send or receive data
  • a "secret", i.e. a string the client will send to authenticate itself
  • how many data connections will be opened
  • the user and group id for who to create the file (in case of send mode)
  • an optional key to encrypt/decrypt the data
• the UFTPD server will now accept an incoming connection from the announced IP address, provided the supplied "secret" matches the expectation.
• if everything is OK, the requested number of data connections from the client can be opened. Firewall transversal will be negotiated using a pseudo FTP protocol.
• the file is sent/received using the requested number of data connections
• to access the requested file, uftpd attempts to switch its user id to the requested one prior to reading/writing the file. This uses a C library which is accessed from Java via the Java native interface (JNI). See also the installation section below.

UFTP client

Usage at HLRS

Since UFTP does not support 3rd party transfer, you need to have console access to a system where either the source or destination filesystem is directly mounted/accessible.

Possible scenarios

Upload/download from/to a site