Instruction for running HEG on Command line mode (batch job) for a bunch of similar files

If you have lots of similar hdf files that need to be processed by HEG, you can use a script similar to the one given below (for example the one provided in HEG's bin directory) or just simply use HEG GUI. If you want to run a batch using HEG GUI please read section 8 of the GUI "Help" HEGBatchProcessing.html

Please follow the instructions below (or the one provided in HEGBatchProcessing.html ) for Unix/Linux or Windows depending on your system.

 

Unix/Linux:

 

a. Create a directory and put all of your similar hdf files (for the same product) in that directory

b. Run HEG GUI so that you create a template parameter file

c. Select File -> Open and choose one of the file in the directory created in step a; click open

d. Select Field, and then click on "down" arrow.

e. Select output file type, projection, etc and click "Accept"

f. Click "Save". Choose your parameter file name in the popped up window and then click "Save" on it. Please note that HEG will append "_sawth " to the end of the name that you chose for your parameter file to indicate that the products that you intended to convert are SWATH type. It will append "_grid" for grid products.

g. Close HEG GUI. Now edit your created parameter file and make the following changes:

  

        1. Change input file name to "input"

        2. change the bounding box (corner lats/lons) to:

 

               SPATIAL_SUBSET_UL_CORNER = ( 90.0 -180.0 )

               SPATIAL_SUBSET_LR_CORNER = ( -90.0 180.0)

 

        3. change your output file name to "input_out" with appropriate            

        extension ".tif" or ".hdf"

        4. save and close this file and rename it to heg_template_unix.prm

 

        Following is an example of such a template file created for a MODIS product:

 

        NUM_RUNS = 1

        BEGIN

        INPUT_FILENAME= /home/abe/GIS_Proj/HEG_MODIS/2007/elt_season/input

        OBJECT_NAME = MOD_Grid_Snow_500m

        FIELD_NAME = Fractional_Snow_Cover

        BAND_NUMBER = 1

        SPATIAL_SUBSET_UL_CORNER = ( 90.0 -180.0 )

        SPATIAL_SUBSET_LR_CORNER = ( -90.0 180.0)

        RESAMPLING_TYPE = NN

        OUTPUT_PROJECTION_TYPE = UTM

        ELLIPSOID_CODE = WGS84

        UTM_ZONE = 0

        OUTPUT_PROJECTION_PARAMETERS = ( 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 )

        OUTPUT_PIXEL_SIZE = 500.0

        OUTPUT_FILENAME =      /home/abe/GIS_Proj/HEG_MODIS/2007/out/melt_season/input_out.tif

        OUTPut_TYPE = GEO

        END

 

 

 

 

h. Create a Shell Script file using following lines and make it executable type (e.g use "chmod 0755 myScript.sh"):

 

 

 

        #! /bin/sh

 

        # in this script it is assumed that HEG executables are in

        # "/home/user/heg2.9/HEG/bin" directory, and that we are

        # processing a file that contains GRIDS. For processing files

        # that contain SWATH use "swtif" instead of "resample"

 

        for i in  `find .` ; do

 

        hdffile=`file $i | awk '{print $2}'`

 

        if [ "$hdffile" = "Hierarchical" ] ; then

 

        sed 's/input/'$i'/' heg_template_unix.prm > $i.prm

        /home/username/heg2.9/HEG/bin/resample -p $i.prm

        fi

        done

 

 

 

i. Before running the script set 3 environment variables (assuming that HEG is installed

        in /home/username/heg2.9/HEG/ ):

 

        setenv MRTDATADIR /home/username/heg2.9/HEG/data

        setenv PGSHOME /home/username/heg2.9/HEG/TOOLKIT_MTD

        setenv HEGUSER ME

 

 

j. Now run the shell script for a HEG batch job!

 

 

 

 

Windows:

 

a. Create a directory and put all of your similar hdf files in that directory

b. Run HEG GUI to create a template parameter file

c. Select File -> Open and choose one of the file in the directory created in step a; click open

d. Select Field, and then click on "down" arrow.

e. Select output file type, projection, etc and click "Accept"

f. Click "Save". Choose you parameter file name in the popped up window and then click "Save" on it. Please note that HEG will append "_sawth " to the end of the name that you chose for your parameter file to indicate that the products that you intended to convert are SWATH type. It will append "_grid" for grid products.

g. Close HEG GUI. Now edit your created parameter file and make the following changes:

a. Change input file name to "input"

b. change the bounding box (corner lats/lons) to:

 

               SPATIAL_SUBSET_UL_CORNER = ( 90.0 -180.0 )

               SPATIAL_SUBSET_LR_CORNER = ( -90.0 180.0)

 

c. change your output file name to "input_out" with appropriate extension ".tif" or ".hdf"

d. save and close this file and rename it to heg_template_unix.prm

 

        Following is an example of such a template file created for a MODIS product:

 

        NUM_RUNS = 1

        BEGIN

        INPUT_FILENAME=C:\GIS_Projects\HEG_MODIS\2007\melt_season\input

        OBJECT_NAME = MOD_Grid_Snow_500m

        FIELD_NAME = Fractional_Snow_Cover

        BAND_NUMBER = 1

        SPATIAL_SUBSET_UL_CORNER = ( 90.0 -180.0 )

        SPATIAL_SUBSET_LR_CORNER = ( -90.0 180.0)

        RESAMPLING_TYPE = NN

        OUTPUT_PROJECTION_TYPE = UTM

        ELLIPSOID_CODE = WGS84

        UTM_ZONE = 0

        OUTPUT_PROJECTION_PARAMETERS = ( 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 )

        OUTPUT_PIXEL_SIZE = 500.0

        OUTPUT_FILENAME = C:\GIS_Projects\HEG_MODIS\2007\melt_season\input_out.tif

        OUTPut_TYPE = GEO

        END

 

 

 

 

Warning:

        Please be aware that in windows if you edit your file with some editors such as wordpad the editor puts carriage return "^M" at the end of each line. This will prevent HEG from running and will cause error. Make sure that your editor will work like UNIX editors. If you have linux or cygwin, do editing there. Also the linux application "dos2unix" cleans  up "^M" from a file.

 

h. The script MyHEG_batchScript.bat in BatchRunning.zip or the one in HEG's bin directory uses MS DOS script commands to run a batch job for HEG in windows. First it edits a template parameter file (created using HEG GUI or by other means as instructed above) to the desired format. The edited file that the script creates, using the .bat file for filename substitution, will contain the LineFeed characters at the end of each line. Since HEG does not recognize this character (causing error) they should be removed. In future this will be worked out to be done with MS DOS commands if possible.  Currently script uses CYGWIN's dos2unix.exe tool that does cleaning for us.  To use this executable we require to have a few other cygwin's  dll files in the same directory that dos2unix.exe exist.

Please also note that if you use MS wordpad for creating parameter file, you can use this executable on command line to get rid of LineFeed characters in your .prm file.

 

Here are the steps for using our Windows batch script:

1.     Download BatchRunning.zip or use those in HEG's bin directory

2.     The sample template parameter file is created for an ASTER product. You need to create your own template parameter file possibly by using HEG GUI, and editing it to put in the desired form as the sample (i.e. having “infile” for the input filename, and in output filename, and modifying the subset lat/lon as instruvted in step b above.)

3.     Extract the files from BatchRunning.zip. In the main directory you will have all executables (except for the resample.exe,  swtif.exe, etc. needed for HEG executable), scripts and dll files

4.     In this directory we have indir and outdir. You should put all your input files in indir. The outdir will contain your template parameter file. After executing the MyHEG_batchScript.bat file your output files will go to outdir directory.

5.     Outputs will have swath or grid area in the original file although in the template parameter file you may use a global subset area.

6.       The Modified parameter files will exist in the main directory for later reference

 

 

 

       Updated November 24, 2015