Skip to content

Driver files

Jump to bottom
Adam Povey edited this page Jun 20, 2022 · 2 revisions

Users are recommended to use the Python scripts to use the ORAC suite. This page provides a summary of the driver files generated by those scripts to control ORAC. It is recommended for experienced users only.

Pre-processor

The following lines are mandatory in the driver file (in this order, without comments):

  1. Name of the sensor. Valid options are AATSR, ABI, AHI, ATSR2, AVHRR, MODIS, SLSTR,SEVIRI and VIIRS.
  2. Path of the level 1B file to be opened.
  3. Path of the geolocation information for that file. For many instruments (such as ATSR2 and geostationary sensors), this field will be ignored.
  4. Path of the USGS land-use and digital elevation model.
  5. Directory of the meteorological file appropriate to this scene. ORAC supports ERA-Interim, ERA5 and output from the ECMWF or GFS operational forecasts. [Note: When using BADC data, http://badc.nerc.ac.uk/data/ecmwf-era-interim, this is specifically the path to the GGAM file.]
  6. Directory containing the RTTOV coefficient file for this sensor. If not with your installation of RTTOV you can download coefficient files from http://www.nwpsaf.eu/site/software/rttov/download/coefficients/coefficient-download.
  7. Directory containing the RTTOV 0.5 degree HSR emissivity data, which can be found at https://www.nwpsaf.eu/site/software/rttov/download/.
  8. Directory containing the NISE snow/sea-ice map appropriate to this scene, which can be found at https://n5eil01u.ecs.nsidc.org/SAN/OTHR/.
  9. Directory containing the MODIS MCD43C3 surface albedo file appropriate to this scene, which can be found at http://ladsweb.nascom.nasa.gov or https://lpdaac.usgs.gov/dataset_discovery/modis/modis_products_table/mcd43c1_v006.
  10. Directory containing the MODIS MCD43C1 surface BRDF file appropriate to this scene, which can be found at the same address as above.
  11. Directory containing the MODIS emissivity map (named global_emis_inf10_monthFilled_MYD11C3), which can also be found at the same address as above.
  12. Reciprocal of longitude for meteorological data grid resolution. This need not be the same as the actual resolution of your met data.
  13. Reciprocal of latitude grid meteorological data resolution.
  14. Directory to save the output files in.
  15. The smallest pixel number to accept in the across-track direction. (A value of 0 in any of the next 4 arguments results in the entire orbit being processed.)
  16. The largest pixel number to accept in the across-track direction.
  17. The smallest pixel number to accept in the along-track direction.
  18. The largest pixel number to accept in the along-track direction.
  19. Version number of NetCDF used.
  20. A comment string specifying the file convention.
  21. A comment string specifying the processing institute.
  22. A comment string specifying the processor used.
  23. A comment string specifying the creator's email address.
  24. A comment string specifying the creator's URL.
  25. A comment string specifying the file version.
  26. A comment string specifying the any references appropriate to this file.
  27. A comment string specifying the file's history.
  28. A comment string specifying a summary of the file.
  29. A comment string specifying any important keywords describing the file.
  30. A comment string specifying any comments about the file.
  31. A comment string specifying the project name. This is the prefix of the output file name.
  32. A comment string specifying the data's license.
  33. A UUID for this file.
  34. The time at which the file was generated.
  35. Path of the AATSR drift correction, which can be found at https://groups.physics.ox.ac.uk/eodg/orac/AATSR_VIS_DRIFT_V03-00.DAT.
  36. Indicate formatting of meteorological data.
    • 0 A single ERA Interim GRIB file, as would be downloaded directly from ECMWF https://apps.ecmwf.int/datasets/data/interim-full-daily/levtype=sfc/.
    • 1 Three ERA Interim NCDF files, preprocessed by RAL from the JASMIN repo.
    • 2 Three ERA Interim files (1 NCDF, 2 GRIB) as found on JASMIN (/badc/ecmwf-era-interim/data).
    • 3 A single ERA Interim NCDF file, pre-processed by RAL from an ECMWF download.
    • 4 A single operational ECMWF NCDF file.
    • 5 A single ERA5 NCDF file.
    • 6 A single GFS4 forecast file.
    • 7 A single GFS reanalysis GRIB file.
    • 8 A single GFS reanalysis NCDF file.
  37. When using three ECMWF files, the GGAS file. Otherwise, ignored
  38. When using three ECMWF files, the SPAM or GPAM file. Otherwise, ignored
  39. True = split the input file into 4096 line chunks; False = Process the entire input as one.
  40. 1 = Process only daytime; 2 = Process only night; 0 or 3 = Process everything.
  41. False = Print nothing to stdout; True = Make verbose output.
  42. This line has been depreciated and is ignored.
  43. True = Assume all directories assigned above actually are full paths to files; False = When passed a directory, ORAC searches it for an appropriate file.
  44. False = Assume a Lambertian surface; True = Use the full BRDF surface treatment.
  45. Version number of RTTOV used.
  46. Version number of ECMWF data used.
  47. Version number of SVN used.

The following are optional arguments. The label is separated from its values by an =:

  • N_CHANNELS) The number of channels to use.
  • CHANNEL_IDS) Paired with N_CHANNELS, this specifies which channels to read from the satellite data.
  • PRODUCT_NAME) Prefix for the output filenames.
  • ECMWF_TIME_INT_METHOD) 2 = Interpolate between two ECMWF files to determine the meteorology; Any other number = Use the meteorology from the nearest ECMWF file to this orbit.
  • ECMWF_PATH_2) When using ECMWF_TIME_INT_METHOD=2, this is the directory of the second GGAM file.
  • ECMWF_PATH2_2) When using ECMWF_TIME_INT_METHOD=2, this is the directory of the second GGAS file.
  • ECMWF_PATH3_2) When using ECMWF_TIME_INT_METHOD=2, this is the directory of the second SPAM file.
  • USE_HR_ECMWF) True = Read a second, higher resolution ECMWF file for better coverage of the surface; False = Don't.
  • ECMWF_PATH_HR) Specifies the directory of the high resolution ECMWF file.
  • ECMWF_PATH_HR_2) When using ECMWF_TIME_INT_METHOD=2, this is the directory of the second high resolution ECMWF file.
  • USE_ECMWF_SNOW_AND_ICE) True = Use the snow/ice field in the ECMWF file; False = Use the NISE map.
  • USE_MODIS_EMIS_IN_RTTOV) True = Use the MODIS emissivity retrieval; False = Use the RTTOV emissivity atlas.
  • ECMWF_NLEVELS) The number of levels in the ECMWF data. Valid values are 60, 91, 137.
  • USE_L1_LAND_MASK) True = Use the land/sea mask from the imager data; False = Use the USGS land/sea mask.
  • USE_OCCCI) True = Use the Ocean Colour CCI product; False = Assume a constant sea-surface backscatter and extinction.
  • OCCCI_PATH) Directory of the Ocean Colour CCI product.
  • USE_PREDEF_LSM) For geostationary sensors: True = Use a predefined land/sea mask stored in an external file (faster). False = ORAC will compute the land/sea mask internally (slower). Do NOT set this to True for non-geostationary instruments!
  • EXT_LSM_PATH) The full filename (including path) to the external land/sea mask used above.
  • USE_PREDEF_GEO) For geostationary sensors: True = Load latitude and longitude from an external file (faster). False = ORAC will, if possible, compute the lat/lon values (slower).Do NOT set this to True for non-geostationary instruments!
  • EXT_GEO_PATH) The full filename (including path) to the external lat/lon file used above.
  • DISABLE_SNOW_ICE_CORR) True = Do not correct surface albedos based upon the snow/ice input dataset. False = Correct albedo for the presence of snow, this can result in blocky output in polar regions.
  • DO_CLOUD_EMIS) Contact Simon Proud for more details ([email protected])
  • DO_IRONLY) True = Force the cloud detection routines to only use infrared channels. False = Use all available channels for cloud detection.
  • DO_CLDTYPE) True = Perform cloud typing. False = Do not perform cloud typing.
  • USE_CAMEL_EMIS) True = Use the CAMEL emissivity database. False = Use the default emissivity database (MYD11).
  • USE_GSICS) SEVIRI only. True [default] = Apply GSICS calibration coefficients to the raw data. Strongly recommended. False = Do not apply GCSICS calibration.
  • USE_CO2) True = Allow variable CO2 in the RTTOV profiles, value is scaled to scene date/time. False = Use single RTTOV CO2 value, fixed to mid 2000's CO2 amount.
  • USE_SWANSEA_CLIMATOLOGY) By default, the pre-processor interpolates the MODIS surface reflectance product to generate BRDF fields. When this parameter is True, it instead reads a climatology of the Swansea surface parameters to produce those fields. Those files can be found at https://groups.physics.ox.ac.uk/eodg/orac/swansea_cb/. This setting requires ASSUME_FULL_PATHS and uses argument 9 of the driver file.
  • SWANSEA_GAMMA) In the calculation for USE_SWANSEA_CLIMATOLOGY, a value must be assumed for the parameter gamma as set here. The default is 0.3.

After running the script, eleven NetCDF files should be produced in the directory specified by line 14.

Main processor

The behaviour of the retrieval is controlled by numerous variables and switches in the Control structure. As such, the main processor has a more advanced driver file parser which can set any variable in that structure. The syntax of the driver mimics standard Fortran.

To illustrate, consider this skeleton (non-functional) driver file.

# ORAC New Driver File
Ctrl%FID%Data_Dir          = /home/me/data
Ctrl%Run_ID                = "Super Awesome Data File"
# This is a comment
Ctrl%Verbose               = True
Ctrl%RS%Use_Full_BRDF      = f
Ctrl%Ind%NAvail            = 14 # This is also a comment
Ctrl%Ind%Channel_Proc_Flag = 1,1,1,1,0,0,0,1,1,1,1,0,0,0
Ctrl%X0[1:3]               = 1.0, -0.5, 300
Ctrl%XB[ITau]              = -1.0
Ctrl%XB[IRe]               = -0.848
  • The first line must take exactly this form. (It indicates the new driver file format should be used.)
  • Each line is broadly the name of the field within the structure to set, an equals sign, and the value it should take.
  • A # indicates that all subsequent characters on that line are comments and should be ignored.
  • Quotes "" delimit a string (and escapes any characters within).
  • Whitespace is not relevant; new lines are.
  • Logical variables, such as Ctrl%Verbose, can be set true with any string starting in T, t, Y, y, or 1; False is F, f, N, n, or 0.
  • An array variable, like Ctrl%Ind%Channel_Proc_Flag, can be set by a comma-separated list. A second array dimension is delimited with a semi-colon ;.
  • You may also assign subsets of an array. Ctrl%X0[1:3] indicates the first to third elements of the array. Other valid indices are
    • [:] the entire array;
    • [:3] the first to third element;
    • [3:] the third to last element;
    • [3] the third element.
  • Various keywords are defined in OracConstants_m, which can be used to improve legibility. For example, Ctrl%XB[ITau] sets the optical depth element of the a priori state vector.

The following fields are mandatory:

  • # ORAC Driver File) This must be the first line.
  • Ctrl%FID%Data_Dir) The directory containing the input files.
  • Ctrl%FID%Filename) The root name of the input files.
  • Ctrl%FID%Out_Dir) The directory into which the outputs should be saved.
  • Ctrl%FID%SAD_Dir) The directory containing the SAD files.
  • Ctrl%!InstName) The name of the instrument to process. For MODIS and AVHRR, this must specify both the instrument and the platform separated by a hyphen (e.g. MODIS-AQUA).
  • Ctrl%Ind%NAvail) The number of channels available in the input files.
  • Ctrl%Ind%Channel_Proc_Flag) A Boolean array specifying which of the available channels should be used during processing.
  • Ctrl%LUTClass) The name of the look-up table to use during processing. Choices include:
Name Description Name Description
WAT Liquid cloud ICE Ice cloud
A70 Dust A71 Polluted dust
A72 Light polluted dust A73 Light dust
A74 Light clean dust A75 NHemisphere background
A76 Clean maritime A77 Dirty maritime
A78 Polluted maritime A79 Smoke

Troubleshooting

  • If you are receiving an error on a line with a string or filename, try surrounding it in "". Some characters (like a full stop) are meaningful to the software and need escaping.
  • If trying to set an allocatable array, make sure it's length is declared before it's contents. In the above, must be set before .
  • The error 'flex parser jammed' probably means you misspelled the name of a variable or forgot an equals, comma, or semi-colon. (At some point these errors will be made more useful.)
  • Setting tau_chans, r_e_chans, ir_chans, or ReChans from the driver file isn't really supported. The issue is that the parser needs to know how long these arrays are, but there is no variable in Ctrl to specify it. Thus, I assume them to be of length 3, 2, 3, and 2 respectively. This, obviously, is rather limiting. I didn't worry about this as these variables, in theory, are hard-coded properties of the instruments.

After running, two NetCDF files should be produced.

Post-processor

The following lines are mandatory in the driver file for cloud-only processing (in this order, without comments):

  1. Full path to the WAT primary file.
  2. Full path to the ICE primary file.
  3. Full path to the WAT secondary file.
  4. Full path to the ICE secondary file.
  5. Full path for the output primary file.
  6. Full path for the output secondary file.
  7. True = Check if the cloud top temperature is consistent with the cloud phase selected; False = Make the cloud phase selection based upon the Pavolonis typing only.

When processing aerosol (with or without cloud), the same arguments are necessary but WAT and ICE should be replaced with the first and second phases evaluated (the ordering of phases isn't relevant). For each additional phase desired, add the full path to the primary and secondary files on adjacent lines. Also, add the following,

USE_BAYESIAN_SELECTION = True

This argument can also be used in cloud processing if you wish to make phase selection based on retrieval cost rather than the Pavolonis typing.

The following are optional arguments. The label is separated from its values by an =:

  • COST_THRESH) When using Bayesian selection, a phase must have greater cost than this limit to be accepted. The default is 0.
  • NORM_PROB_THRESH) When using Bayesian selection, the cost is used to calculate a probability for each phase. To be selected, that must be greater than this limit to be accepted.
  • OUTPUT_OPTICAL_PROPS_AT_NIGHT) True = Output optical depth and effective radius at night; False = Don't.
  • VERBOSE) False = Print nothing to stdout; True = Make verbose output.
  • USE_CHUNKING) True = Split the input file into 4096 line chunks to save memory; False = Process the entire input as one.
  • USE_NETCDF_COMPRESSION) True = Compress the output files; False = Don't.

Clone this wiki locally