Lustre short read

Short reads.

If you are trying to read a certain amount of data (i.e. 1 MB) the POSIX standard allows to return the read with less data read than actually requested (i.e. 500 kB). This is called a short read. The read command will return the length of the record read.

It is the programmers responsiblity to check if the application actually read the amount data requested. If less data have been read this should be handled by the programm i.e. by re-reading the missing data.

The following example ensures that all data requested has been read:

   read:
   if(len > 0) {                                                      //len length of the array that is being read.
       count = fread(addr, (size_t)len, (size_t)1, (FILE*) data);     //addr buffer, where the data will be read to 
                                                                      //fread returns the number of bytes read
       if((count<len && count>0 ) || (count==-1 && errno==EINTR)){     
           addr+=count;
           len-=count;
           goto read;
       }
   }

Of course you should also check for reading errors.

Short reads and Other IO libraries

The following I/O libraries handle short reads in a proper way:

• MPIIO
• HDF5 parallel
• NETCDF 4
• Fortran I/O

Issues occur with the following libraries

• C or C++ I/O The programmer has to handle the short read himself
• NETCDF 3 leads to an unwanted behavior: If a short read occurs netcdf will fill up the missing data in the array with zeroes and returs successfully.