- Infos im HLRS Wiki sind nicht rechtsverbindlich und ohne Gewähr -
- Information contained in the HLRS Wiki is not legally binding and HLRS is not responsible for any damages that might result from its use -

Lustre short read: Difference between revisions

From HLRS Platforms
Jump to navigationJump to search
(Created page with "== Short reads == The POSIX IO standard allows short reads. If you are trying to read a certain amount of data (i.e. 1 MB) the POSIX standard allows to return the read with...")
 
Line 13: Line 13:
     if(len > 0) {
     if(len > 0) {
         count = fread(addr, (size_t)len, (size_t)1, (FILE*) data);
         count = fread(addr, (size_t)len, (size_t)1, (FILE*) data);
         if(count<=len){
         if(count<len){
             addr+=count;
             addr+=count;
             len-=count;
             len-=count;
Line 19: Line 19:
         }
         }
     }
     }
 
Of course you should also check for reading errors.
 
 
 


== Short reads and Other IO libraries ==
== Short reads and Other IO libraries ==

Revision as of 11:16, 20 April 2015

Short reads

The POSIX IO standard allows short reads.

If you are trying to read a certain amount of data (i.e. 1 MB) the POSIX standard allows to return the read with less data read than actually requested (i.e. 500 kB). This is called a short read. The read command will return the length of the record read.

It is the programmers responsiblity to check if the application actually read the amount data requested. If less data have been read this should be handled by the programm i.e. by re-reading the missing data.


The following example ensures that all data requested has been read:

   read:
   if(len > 0) {
       count = fread(addr, (size_t)len, (size_t)1, (FILE*) data);
       if(count<len){
           addr+=count;
           len-=count;
           goto read;
       }
   }

Of course you should also check for reading errors.

Short reads and Other IO libraries

The following I/O libraries handle short reads in a proper way:

  • MPIIO
  • HDF5 parallel
  • NETCDF 4
  • Fortran I/O


Issues occur with the following libraries

  • C or C++ I/O The programmer has to handle the short read himself
  • NETCDF 3 leads to an unwanted behavior: If a short read occurs netcdf will fill up the missing data in the array with zeroes and returs successfully.