MODIS Processing has stopped working

[thomas.storm]

Dear all,

we are running an operational service that processes MODIS Aqua and VIIRS data on a daily basis. While VIIRS is running without problems, since March 2nd the geogen_modis process fails with the message:
geogen_modis returned with exit status: 1
Percent valid data (0.00) is less than threshold (95.00)

From our experience, this is caused by wrong or missing auxiliary data; but the auxiliary data is present. Has anything changed on the OBPG side in the last days?

Help is much appreciated!

Cheers,
    Thomas

[OB.DAACx - SeanBailey]

Thomas,

No, nothing changed (that I'm aware).  When geogen_modis fails, the script leaves behind several files for diagnosis.
You can force this behavior with the boolean --log option.  There are three files with the pattern Log*.<GEOfile>, LogReport, LogUser and LogStatus.
The LogStatus tends to be the most informative, although generally the information is a bit cryptic.  If after looking at these files
you cannot identify the problem, please attach them to a reply post and we'll see if we can interpret them :grin:

Sean

[thomas.storm]

Sean,

thanks for your answer. I have found in the LogStatus the following:
waqss-modis-process: /data3/swdev/V2016.2/build/src/geogen_modis/GEO_location_main.c, main():MODIS_U_GEO_BEGIN:288778279
waqss-modis-process: Thu Mar  9 04:11:48 2017
waqss-modis-process:  Seed file: 6.2 Running MOD_PR03 MODIS Geolocation main() 6.2 compiled on Apr 26 2016 at 15:06:00 using makefile 1.1
waqss-modis-process:
waqss-modis-process: getMODISfileinfo():MAPI_E_ERR:324431360
waqss-modis-process: ERROR: getMODISfileinfo cannot find file attribute Extract Pixel Offset
waqss-modis-process:
waqss-modis-process:
waqss-modis-process: getMODISfileinfo():MAPI_E_ERR:324431360
waqss-modis-process: ERROR: getMODISfileinfo cannot find file attribute Extract Pixel Count
waqss-modis-process:
waqss-modis-process:
waqss-modis-process: getMODISfileinfo():MAPI_E_ERR:324431360
waqss-modis-process: ERROR: getMODISfileinfo cannot find file attribute Extract Line Offset
waqss-modis-process:
waqss-modis-process:
waqss-modis-process: getMODISfileinfo():MAPI_E_ERR:324431360
waqss-modis-process: ERROR: getMODISfileinfo cannot find file attribute Extract Line Count

Also, there is a vast number of lines like this:

waqss-modis-process: PGS_TD_TAItoUT1pole():PGSTD_E_NO_UT1_VALUE:28176
waqss-modis-process: no UT1-UTC or polar motion available for input UTC time, assuming UT1-UTC and polar motion values are 0


followed by these error messages:

waqss-modis-process: /data3/swdev/V2016.2/build/src/geogen_modis/GEO_get_T_inst2ecr.c, GEO_get_T_inst2ecr():MODIS_E_GEO:288779784
waqss-modis-process: Thu Mar  9 04:07:59 2017
waqss-modis-process: Error returned by function PGS_CSC_ECItoECR(2017-03-08T06:54:59.553997Z)
waqss-modis-process:
waqss-modis-process: /data3/swdev/V2016.2/build/src/geogen_modis/GEO_interp_ECR.c, GEO_interp_ECR():MODIS_E_GEO:288779784
waqss-modis-process: Thu Mar  9 04:07:59 2017
waqss-modis-process: Error returned by function GEO_get_T_inst2ecr(2017-03-08T06:54:59.553997Z)
waqss-modis-process:
waqss-modis-process: /data3/swdev/V2016.2/build/src/geogen_modis/GEO_solar_and_lunar_vectors.c, GEO_solar_and_lunar_vectors():MODIS_E_GEO:288779784
waqss-modis-process: Thu Mar  9 04:07:59 2017
waqss-modis-process: Error returned by function GEO_interp_ECR(763109709.554)
waqss-modis-process:
waqss-modis-process: PGS_CSC_UTC_UT1Pole():PGSCSC_W_JD_OUT_OF_RANGE:35840
waqss-modis-process: no UT1-UTC or polar motion available for input UTC time, assuming UT1-UTC and polar motion values are 0
waqss-modis-process:
waqss-modis-process: /data3/swdev/V2016.2/build/src/geogen_modis/GEO_get_utcpole_metadata.c, GEO_get_utcpole_metadata():MODIS_E_GEO:288779784
waqss-modis-process: Thu Mar  9 04:07:59 2017
waqss-modis-process: Error returned by function PGS_CSC_UTC_UT1Pole(2017-03-08T06:50:00.671298)
waqss-modis-process:
waqss-modis-process: /data3/swdev/V2016.2/build/src/geogen_modis/GEO_write_granule_metadata.c, GEO_write_granule_metadata():MODIS_E_GEO:288779784
waqss-modis-process: Thu Mar  9 04:07:59 2017
waqss-modis-process: Error returned by function GEO_get_utcpole_metadata()
waqss-modis-process:
waqss-modis-process: GEO_get_GRing_points.c, GEO_get_GRing_points():MODIS_W_NO_GEO:288779311
waqss-modis-process: Thu Mar  9 04:07:59 2017
waqss-modis-process: Granule contains no geolocatable pixels.
waqss-modis-process:
waqss-modis-process: /data3/swdev/V2016.2/build/src/geogen_modis/GEO_locate_one_granule.c, GEO_locate_one_granule():MODIS_E_GEO:288779784
waqss-modis-process: Thu Mar  9 04:07:59 2017
waqss-modis-process: Error returned by function GEO_write_granule_metadata("/hd4/yarn/local/usercache/cvop/appcache/application_1487056014494_17705/container_1487056014494_17705_01_000029/A2017067065000.GEO","/hd4/yarn/loc
al/usercache/cvop/appcache/application_1487056014494_17705/container_1487056014494_17705_01_000029/A2017067065000.L1A_LAC")
waqss-modis-process:
waqss-modis-process: /data3/swdev/V2016.2/build/src/geogen_modis/GEO_location_main.c, main():MODIS_E_GEO:288779784
waqss-modis-process: Thu Mar  9 04:07:59 2017
waqss-modis-process: Error returned by function GEO_locate_one_granule(/hd4/yarn/local/usercache/cvop/appcache/application_1487056014494_17705/container_1487056014494_17705_01_000029/A2017067065000.L1A_LAC,/hd4/yarn/local/usercache/cvop/
appcache/application_1487056014494_17705/container_1487056014494_17705_01_000029/A2017067065000.GEO)
waqss-modis-process:
waqss-modis-process: /data3/swdev/V2016.2/build/src/geogen_modis/GEO_location_main.c, main():MODIS_U_GEO_GRANULE_ID:288778281
waqss-modis-process: Thu Mar  9 04:07:59 2017
waqss-modis-process: GRANULE_ID:/hd4/yarn/local/usercache/cvop/appcache/application_1487056014494_17705/container_1487056014494_17705_01_000029/A2017067065000.L1A_LAC
waqss-modis-process:
waqss-modis-process: /data3/swdev/V2016.2/build/src/geogen_modis/GEO_location_main.c, main():MODIS_U_GEO_END:288778280
waqss-modis-process: Thu Mar  9 04:07:59 2017
waqss-modis-process: MOD_PR03 controlled exit, with exit code  = 1

This looks like it may have caused the process to file, but I am not sure why this happens now. Do you have any ideas?

Cheers,
    Thomas

[OB.DAACx - SeanBailey]

Looks like your polar wander data are out of date:

> no UT1-UTC or polar motion available for input UTC time, assuming UT1-UTC and polar motion values are 0

run update_luts.py aqua, or manually pull down the utcpole.dat file
and put it in your $OCSSWROOT/run/var/modis/ directory.

Sean

[thomas.storm]

Sean,

sorry for answering that late -- I did the manual update, and the process runs successfully again. Thanks!

Cheers,
    Thomas