NEC SX ACE Disk Storage

There is no backup of data. Most important first.

Home filesystem has quotas active and is shared with other HWW systems. Temporary diskspace shared between frontend and SX-ACE nodes can only be requested through workspace mechanism.

For home and workspace filesystems, best performance is achieved with large I/O in blocks > 4MB. All filestytems can be accessed from SXes and frontends under same path.

Avoid formated I/O on SX, try to do large block, unformated I/O.

You can use the variables F_SETBUF and C_SETBUF to set buffer size of standard buffered I/O.

There is NO LOCAL disk space on SX-ACE nodes!

To copy large files on SX use

cp -b 32768 source target

this speeds up the copy by a factor of 5 vs. the default (cp -b 1024), a buffer > 32768 will not work!

Tuning IO on ScaTeFS

The ScaTeFS configuration consists of 2 servers and 18 storage targets on 4 RAID units. The filesystems knows two ways of storing files, striped and unstriped.

An example for an unstriped file:

$ scatefs_getfinfo testfile
format     : non stripe format
iot count  :                18
stripesize :         268435456
chunksize  :         268435456
filesize   :       10737418240

An example for a striped file:

$ scatefs_getfinfo testfile
format     :     stripe format
iot count  :                18
stripesize :           4194304
chunksize  :        1073741824
filesize   :       10737418240

Default is to write unstriped files.

You can change this in two ways, by setting an attribute to the directory before creating files, or by setting the attributes for the file before writing to it.

Example how to set a stripesize of 4MB for all files in a directory

$ scatefs_setdirattr -s 4m testdir
$ touch testdir/testfile
$ scatefs_getfinfo testdir/testfile
format     :     stripe format
iot count  :                18
stripesize :           4194304
chunksize  :        1073741824
filesize   :                 0

Example how to set a stripesize of 1MB for a file in a directory

$ scatefs_premap -s 1M 0 testdir/testfile1m
$ scatefs_getfinfo testdir/testfile1m
format     :     stripe format
iot count  :                18
stripesize :           1048576
chunksize  :        1073741824
filesize   :                 0

The 0 in front of filename denotes the file size, if the filesize to be written is known, some information can be preallocated to speedup later processing, if not known, give 0.

On SX-ACE, a stripesize of 4M with an IO size of 16MB gives something like 700MB/s write bandwidth into the filesystem. IO size should be multiple of stripsize, and 4M seems to be a good value for stripsize.