aug_eos5 2.2 Description ~ This package contains a program that augments an existing HDF-EOS5 file so that the netCDF-4 library may read. This program mainly adds HDF5 dimension scales with actual dimensional values, which can be recognized by the netCDF-4 library, and associates them with corresponding HDF5 datasets. We provide an option for users to use an input file to provide missing dimension and dimension scale information. Limitation ~ We are assuming that all HDF5 datasets are accessible via the HDF-EOS5 API. This limitation is caused by the fact that we use the HDF-EOS5 API to collect all information. If an HDF-EOS5 file contains an HDF5 dataset that is invisible by the HDF-EOS5 API, this program does not add dimension scales for that dataset, which may cause the netCDF-4 library to fail to read the file even after augmentation. Also, we assume that all object names used in existing HDF-EOS5 file are allowed in the netCDF-4 library. Requirement ~ HDF-EOS5 library (1.12 or higher) HDF5 library (1.6 or higher) After augmentation, the HDF-EOS5 API will say that every data field has an attribute named DIMENSION_LIST. If any program tries to access this dimension, the HDF-EOS5 library will return a failure. This is caused because DIMENSION_LIST is a pre-defined attribute used for implementing HDF5 dimension scale. We are expecting the next HDF-EOS5 release will detect that special attribute and the API will not expose that attribute to users. This augmentation program does not use netCDF-4 at all, but you may want to check if this file is readable. Files ~ Source code of aug_eos5: cvct.h read.h util.h write.h main.c read.c util.c write.c NetCDF4 testing programs for reading augmented files: check.f90 check_za.f90 check.c Sample HDF-EOS5 file: example.he5 example_za.he5 Mapping file for instructing augmentation tool how to handle dimension: dimgeo.txt dimgeo_za.txt Sample makefiles makefile.augment makefile.check Install ~ 1. After uncompressing the package tarball, execute configure script with appropriate path names. For example, $ ./configure \ --with-zlib=/home/packages/zlib-123/Linux2.6-gcc \ --with-szlib=/mnt/scr1/clee83/sysdep/kagiso/szip_encdec/szip \ --with-hdf5=/home/packages/hdf5/v167/Linux_2.6/standard \ --with-hdfeos5=/mnt/scr1/clee83/sysdep/jam/hdfeos5_he_ue/hdfeos \ If you are using HDF5 1.8 or higher, you need to add CPPFLAGS=-DH5_USE_16_API to the above command. 2. Build $ make This will generate the executable file that augments the HDF-EOS5 file to be accessible by netCDF-4 APIs. The executable file is called aug_eos5 and is located under the src directory. or $ make -f makefile.augment If ./configure does not work on your machine, you could generate the executable file by using the makefile template we provided. The executable file is the same as the one generated by using ./configure. For makefile.augment to work properly, you should modify the library and include paths based on your own system configuration. Note: For Mac users, ignoring the warnings like: run.c: In function 'gridaugalloc_info': run.c:56: warning: format '%4d' expects type 'int', but argument 3 has type 'size_t' run.c:56: warning: format '%4d' expects type 'int', but argument 4 has type 'size_t' The executable will work properly. $ make -f makefile.check This will generate three exectutables, which are used to check if netcdf-4 C and Fortran APIs can access the augmented files as expected. Their names are check_c, check_f and check_za_f respectively. For makefile.check to work properly, you should modify the library and include paths based on your own system configuration. 3. Usage a. Run aug_eos5 as follows: $ ./aug_eos5 -f dimgeo.txt example.he5 or execute unittest/test.sh example.he5 is the file to be augmented. For this example file, the connection between the dimension of a variable and the corresponding coordinate variable is missing. The dimgeo.txt file will provide the link between the dimension and the coordinate variable. Its content is as follows: 0 nTimes Time 0 nLevels Pressure 2 nCloudTypes nTimes, nLevels and nCloudTypes are the dimension names and Time and Pressure are the corresponding coordinate variables. To find more information about the concept of and the usage of the mapping file, please read the augmentation.txt. Since this program augments the given file, the given file should be writable. b. Result As a result, the given HDF-EOS5 is augmented. Augmented file has new dimension variables which has real values and attributes. c. Reading augmented file using NetCDF4 library. $ ./check_c -t 0 -g HIRDLS -v CFC11 example.he5 or execute unittest/validate_c.sh where t represents type(swath, grid or zonal); g represents group name; and v represents variable name. In this case, check_c may read CFC11 under /HDFEOS/SWATHS/HIRDLS/Data Fields/ in example.he5. $ ./check_c -t 2 -g HIRDLS -v Day HIR3ZA-test2.he5 check_c may read Day under /HDFEOS/ZAS/HIRDLS/Data Fields/Day in HIR3ZA-test2.he5. $ ./check_f or execute unittest/validate_f.sh where check_f checks CFC11 in example.he5 as well. $ ./check_za_f or execute unittest/validate_f.sh where check_za_f checks Day in examle_za.he5. check_c is a C program binary. check_f and check_za_f are fortran90 program binary. They do the same job: open the augmented file and show information about a variable. As with this release, the variable type is limited to Float. Note: There has been no change for check_f since ver 1.2. For check_c, new options are provided in ver 2.0 and 2.2. check_za_f is new add-on in version 2.2. For more information about validation, please read the validation.txt. The augmented file can also use ncdump to check as: $ ncdump -h example.he5 or $ ncdmup -v CFC11 example.he5 The current NetCDF-4.1.1 released on April 8, 2010 may fail to read long fixed-length HDF5 string. So "$ ncdump example.he5" may get segmentation fault . Please see the known problms for more details. 4. Test (optional) This step tests whether netCDF-4 library and ncdump can read augmented files. For proper testing, the directory containing ncdump binary should be included in the PATH env varaible. For test, unaugmented eos5 files need to be downloaded first. This can be done by executing $ cd test $ ./download.sh or for Mac usrs $ ./download_mac.sh After this procedure, the following files can be found under test/data_set_2/ HIRDLS-Aura_L2_v02-04-09-c03_2008d001.he5 MLS-Aura_L2GP-BrO_v02-23-c01_2010d255.he5 MLS-Aura_L2GP-Temperature_v02-23-c01_2010d058.he5 OMI-Aura_L2-OMBRO_2010m1005t1835-o33108_v003-2010m1006t023445.he5 OMI-Aura_L3-OMAEROe_2010m0830_v003-2010m0902t032205.he5 TES-Aura_L3-O3_r0000011812_C01_F01_07.he5 TES-Aura_L2-SUPPLEMENTAL-SO_r0000011860_C01_F01_07.he5 HIR3ZA-test2.he5 GSSTF.2b.2008.01.01.he5 Then, execute test.sh script as follows: $ ./test.sh If any problem is found during the tests, testing procedure will be stopped and details will be shown. Note: test.sh uses h5diff to compare. please make sure the PATH environment is set properly. 5. Products We have tested augmentation tool against HIRDLS, MLS, TES, OMI, and MEaSUREs on 32-bit and 64-bit Linux, as well as 64-bit Mac. 6. Known Problems The current NetCDF-4.1.1 released on April 8, 2010 may fail to read long fixed-length HDF5 string. So dumping all the contents of an augmented file with ncdump("ncdump example.he5") may get segmentation fault. The long fixed-length HDF5 string(such as StructMetadata.0) in an HDF-EOS5 file mainly store some metadata information essential for the HDF-EOS5 applications. So currently we think that the long fixed-length HDF5 string the netCDF-4 APIs fails to read is not important to netCDF-4 users. We already reported this problem to the netCDF-4 team. Hopefully this problem will be fixed in their future release. vim:ft=help