HDF4 CF Conversion Library

The HDF4 CF (H4CF) conversion library can be used to access various types of NASA HDF4external and HDF-EOS2 files by following CF conventions. The CF conventions are defined hereexternal. We have incorporated the knowledge and information obtained from the NASA HDF-EOS2 and HDF4 files and the corresponding documents published by NASA data centers into this conversion library. We also have implemented an HDF4 to netCDF-3 conversion tool that follows CF conventions by using this conversion library. In this web page, we will provide basic usage of the conversion library and the HDF4 to netCDF-3 conversion tool. A set of simple conversion library APIs will be introduced and some screenshots of using netCDF visualization tools to access the converted netCDF files will also be presented.

Obtaining the Software and Installation

Users can download the source code package from this link. Users should unpack the gzipped tar archive file to a location where source packages are usually built locally on your machine. /usr/local/src, for example.

The H4CF conversion library requires the HDF4external , HDF-EOS2 and NetCDFexternal external libraries. HDF4 library must be configured with "--disable-netcdf" option. Otherwise, the HDF4 to netCDF-CF conversion tool may not be compiled or built.

We provide two methods to build the conversion library and the HDF4 to netCDF-3 conversion tool. The first method is to set up the build environment using a configure script, which is maintained by the Autoconfexternal system. The user should pass the paths to the installed location of the prerequisite libraries when running the configure script. Figure 1 shows example configuration options. The uses should tailor the values to reflect the actual locations on the system where the software is being installed.

Figure 1 Configure the build environment
$ ./configure \
--with-prefix= \
--with-jpeg=/usr/local/src/jpeg-6b \
--with-zlib=/usr/local/src/zlib \
--with-szlib=/usr/local/src/szip \
--with-hdf4=/usr/local/src/hdf4 \
--with-hdfeos2=/usr/local/src/hdfeos/hdfeos2 \
--with-netcdf=/usr/local/netcdfbin
If HDF4 and HDF-EOS2 are built without using the szip library, then the --with-szlib option should be omitted. For some systems that have already installed the zlib and the jpeg libraries, --with-jpeg and --with-zlib can also be omitted. In this case, one may use the following:
Figure 2 Configure the build environment without szip, zlib and jpeg
$ ./configure \
--with-hdf4=/usr/local/src/hdf4 \
--with-hdfeos2=/usr/local/src/hdfeos/hdfeos2 \
--with-netcdf=/usr/local/netcdfbin
The above command should succeed and generate a Makefile. With the Makefile, one can then build the H4CF library, the CDL Dumper and the HDF4 to NetCDF Converter utility programs by issuing the make command.
Figure 3 Invoke the make command
$ make CXXFLAGS=-O0
...
$ make install

Typing make install installs the H4CF conversion library and the corresponding tools. BE AWARE: when you build the library, you must type make CXXFLAGS=-O0 to ensure the correctness of the library and the tools. Without using CXXFLAGS=-O0 the library and the tools may NOT be usable.

With the default value of the prefix variable in the above configuration, the make command will produce the standalone static library files: h4cf/lib/libh4cf.a and h4cf/lib/libh4cf.la. The C++ header files to be included by application programs which use the H4CF library are installed into h4cf/include. the utility programs cdldumper and h4tonccf are installed into h4cf/bin.

For the HDF4 to netCDF-CF conversion tool h4tonccf, the above "configure" and "make" commands will generate a version of a tool to produce netCDF-3 files from HDF-EOS input files. To generate netCDF-4 classic files, one should use the following configuration command:

Figure 4 Configure the build environment to convert to netCDF-4 classic files
$ ./configure \
--with-jpeg=/usr/local/src/jpeg-6b \
--with-zlib=/usr/local/src/zlib \
--with-szlib=/usr/local/src/szip \
--with-hdf4=/usr/local/src/hdf4 \
--with-hdfeos2=/usr/local/src/hdfeos2 \
--with-hdf5 = /your hdf5 library path\
--with-netcdf=/your netcdf library path \
--prefix=/home/local
$ make CXXFLAGS=-O0
$ make install
One should be aware that we haven't thoroughly tested the conversion to netCDF-4 classic format. We do find that some files that can be converted to netCDF-3 can not be converted to netCDF-4 classic format.

The second option is to use a manually created Makefile. We have provided three template Makefiles to help users to follow. These Makefile templates are: src/Makefile.template to generate the h4cf conversion library, utility/Makefile.template to generate the hdf4 to netcdf conversion tool and internal_test/Makefile.template to generate the cdldumper debugging tool. The user must edit the value of the directory path to each required library in each Makefile to conform to the system where the software is being installed.

We also provide in the package a directory of examples: read, read_subset, read_var_attrs and read_file_attrs. To build these examples, modify the examples/Makefile.template to conform to the configuration of your system and compile them as follows:

Figure 5 Build the examples
$ cd examples
$ vi Makefile.template
$ make -f Makefile.template

Using the H4CF Library API

HDF4 and HDF-EOS2 groups, variables and attributes are presented under a single, unified abstraction. When a file is opened, a pointer to the root group is returned. The user can traverse all the structures from the root group pointer. If user opens the file with CF option, the structure shown to the user is CF-compliant.

Figure 6 H4CF Library API example
// open the example HDF file
h4cf_open("geo.hdf");

// HDF file attributes are obtained in a C++ STL map object
const map<string, string> file_attrs = h4cf_get_file_attrs();

// HDF file variables are obtained in a C++ STL list object
const list<var*> pvars = h4cf_get_vars();

// close the HDF file
h4cf_close();
The user can traverse the file attribute map using the C++ Standard Template Library mapexternal iterator map<string,string>::const_iterator it=file_attrs.begin() method. The user can traverse the file variable list using the C++ STL listexternal iterator list<var*>::const_iterator var_iter = pvars.begin() method.

The example programs employ these API methods to access the groups of the HDF file. The sample HDF file examples/geo.hdf used in the following examples is included in the source code distribution. The first example, examples/read.cpp prints the value of each variable named "temp" in the file, where the dimensions of the data field can be seen to be XDim=36, YDim=18 by using either the EOS2 Dumperexternal or HDFViewexternal.

Figure 7 H4CF Library read example
$ examples/read
Row: 0
0 1 2 3 4 5 6 7 8 9
10 111213141516171819
20 212223242526272829
30 3132333435
Row: 1
36373839404142434445
46 474849505152535455
56 575859606162636465
66 6768697071
...
Row: 16
576 577 578 579 580 581 582 583 584 585
586587 588 589 590 591 592 593 594 595
596597 598 599 600 601 602 603 604 605
606607 608 609 610 611
Row: 17
612 613 614 615 616 617 618 619 620 621
622623 624 625 626 627 628 629 630 631
632633 634 635 636 637 638 639 640 641
642643 644 645 646 647
The second H4CF Library API example does a subsetted read from the variable called "temp" using grid parameters start = {0,0}, stride = {2,2} and edge = {4,4} and prints the resulting 4-by-4 subsetted grid, which contains every second value from the variable.
Figure 8 H4CF Library read_subset example
$ examples/read_subset
vals[0,0]=0 vals[0,1]=2 vals[0,2]=4 vals[0,3]=6
vals[1,0]=72vals[1,1]=74vals[1,2]=76vals[1,3]=78
vals[2,0]=144 vals[2,1]=146 vals[2,2]=148 vals[2,3]=150
vals[3,0]=216 vals[3,1]=218 vals[3,2]=220 vals[3,3]=222
The third H4CF Library API example reads the attributes from the variable named "temp" and prints name of the attribute called "_FillValue" along with the value of that attribute.
Figure 9 H4CF Library read_var_attrs example
$ examples/read_var_attrs
_FillValue = -999
This final H4CF Library API example prints the contents of the C++ STL map object obtained by the h4cf_get_file_attrs() method.
Figure 10 H4CF Library read_file_attrs example
$ examples/read_file_attrs
HDFEOSVersion = HDFEOS_V2.16StructMetadata_0 = GROUP=SwathStructure
END_GROUP=SwathStructure
GROUP=GridStructure
GROUP=GRID_1
GridName="grid1"
XDim=36
YDim=18
UpperLeftPointMtrs=(-180000000.000000,90000000.000000)
LowerRightMtrs=(180000000.000000,-90000000.000000)
Projection=GCTP_GEO
GROUP=Dimension
END_GROUP=Dimension
GROUP=DataField
OBJECT=DataField_1
DataFieldName="temp"
DataType=DFNT_FLOAT64
DimList=("YDim","XDim")
END_OBJECT=DataField_1
END_GROUP=DataField
GROUP=MergedFields
END_GROUP=MergedFields
END_GROUP=GRID_1
END_GROUP=GridStructure
GROUP=PointStructure
END_GROUP=PointStructure
END

Using the H4CF Utility Programs

CDL Dumper usage is as follows:

Figure 11 CDL Dumper usage
$ h4cf/bin/cdldumper
Usage: cdldumper [-H] [hdf file name]
Attention: If '-H' is specified, only header part is dumped.
The following is an example of using the CDL Dumper to access an HDF-EOS2 file from the National Snow and Ice Data Center (NSIDC)external. Users can download the HDF file used in this example from hereexternal.

We invoke the CDL Dumper as follows:

Figure 12 CDL Dumper example
$ h4cf/bin/cdldumper AMSR_E_L3_5DaySnow_V09_20050126.hdf
We compare the output from CDL Dumper with what is obtained using HDFViewexternal.

NetCDF Converter usage is as follows:

Figure 14 NetCDF Converter usage
$ h4tonccf
Usage: h4tonccf [hdf file name] or h4tonccf [hdf file name] [netcdf file name]
Note: If netcdf file name is not specified, hdf file name will be used instead with .nc as extension.
We will convert an HDF-EOS2 data file from NASA Goddard Space Flight Center Atmosphere Archive and Distribution System (LAADS)external to NetCDF and examine it using Unidata IDVexternal. Users can download the HDF file used in this example from hereexternal.
Figure 15 Converting LAADS MODIS-AQUA data file to NetCDF
$ h4tonccf MYD021KM.A2002226.0000.005.2009193222735.hdf
Done with writing netcdf file MYD021KM.A2002226.0000.005.2009193222735.nc.
The following is the view of the data fields of the converted NetCDF file MYD021KM when loaded into IDV. The following is the Color-shaded plan view of the Earth View 1KM Emissive Bands data field of file MYD021KM in IDV.

Next, we will convert a MODIS-T file from the NASA Goddard Space Flight Center Ocean Biology Processing Group (OBPG)external to NetCDF and examine it using Panoplyexternal. Users can download the HDF file used in this example from hereexternal.

Figure 18 Converting OBPG HDF data file to NetCDF
$ h4tonccf T20000322000060.L3m_MO_NSST_4.hdf
Done with writing netcdf file T20000322000060.L3m_MO_NSST_4.nc.
The following is the view of the data fields of the converted NetCDF file T20000322000060.L3m_MO_NSST_4.nc when loaded into Panoply. The following is the plot of T20000322000060 l3m_data as generated by Panoply.

Finally, we will convert a file from the NASA Goddard Earth Sciences Data and Information Center (GESDISC)external to NetCDF and examine it using IDV. Users can download the HDF file used in this example from hereexternal.

Figure 21 Converting GESDISC AIRS HDF-EOS2 data file to NetCDF
$ h4tonccf AIRS.2002.08.01.L3.RetStd_H031.v4.0.21.0.G06104133732.hdf
Done with writing netcdf file AIRS.2002.08.01.L3.RetStd_H031.v4.0.21.0.G06104133732.nc
The following is the view of the data fields of the converted NetCDF file AIRS.2002.08.01.L3.RetStd_H031.v4.0.21.0.G06104133732 when loaded into IDV. The following is the Color-Shaded Plan View of the Temperature_WM_A data field of the converted NetCDF file AIRS.2002.08.01.L3.RetStd_H031.v4.0.21.0.G06104133732.nc as generated by IDV.


Last modified: 08/09/2011
About Us | Contact Info | Archive Info | Disclaimer
Sponsored by NASA Cooperative Agreement Grant Number NNX08AO77A / Maintained by The HDF Group