- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

Lustre short read: Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
Line 11: Line 11:
     if(len > 0) {
     if(len > 0) {
         count = fread(addr, (size_t)len, (size_t)1, (FILE*) data);
         count = fread(addr, (size_t)len, (size_t)1, (FILE*) data);
         if(count<len){
         if((count<len && count>0 ) || (count==-1 && errno==EINTR)){
             addr+=count;
             addr+=count;
             len-=count;
             len-=count;
Line 17: Line 17:
         }
         }
     }
     }
Of course you should also check for reading errors.
Of course you should also check for reading errors.



Revision as of 13:39, 6 July 2015

Short reads.

If you are trying to read a certain amount of data (i.e. 1 MB) the POSIX standard allows to return the read with less data read than actually requested (i.e. 500 kB). This is called a short read. The read command will return the length of the record read.

It is the programmers responsiblity to check if the application actually read the amount data requested. If less data have been read this should be handled by the programm i.e. by re-reading the missing data.


The following example ensures that all data requested has been read:

   read:
   if(len > 0) {
       count = fread(addr, (size_t)len, (size_t)1, (FILE*) data);
       if((count<len && count>0 ) || (count==-1 && errno==EINTR)){
           addr+=count;
           len-=count;
           goto read;
       }
   }


Of course you should also check for reading errors.

Short reads and Other IO libraries

The following I/O libraries handle short reads in a proper way:

  • MPIIO
  • HDF5 parallel
  • NETCDF 4
  • Fortran I/O


Issues occur with the following libraries

  • C or C++ I/O The programmer has to handle the short read himself
  • NETCDF 3 leads to an unwanted behavior: If a short read occurs netcdf will fill up the missing data in the array with zeroes and returs successfully.