Pass processed NRT OK, reprocess 5 days later fails?
Posted: Tue Mar 10, 2020 1:54 pm America/New_York
Sean,Sorry for the long post. But... I ran this combined granule NRT on 2020-03-05 13:53:07 , and it failed as happens when finding a new way to do things. II ran it again at 2020-03-05 15:02:38 (still NRT) and it was was successful. However, I reprocess things days later, and that failed at the place where the NRT processing worked fine. See bolded below:Here is the good processing from 3/5STATUS: $ocssw_command_1 = modis_L1A.py /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.PDS -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC --startnudge=0 --stopnudge=10STATUS: $ocssw_return_1 = MODIS L1A version 6.0.6, built Aug 16 2019 12:46:26scan: 0 out of 404 Thu Mar 5 15:03:06 2020...scan: 390 out of 404 Thu Mar 5 15:03:09 2020STATUS: $l1a_file_w_vdir_path /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC was found$_: att1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1550.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1550.061 was not compressedSTATUS: $compression_test_returns existing file: 1$_: att2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1555.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1555.061 was not compressedSTATUS: $compression_test_returns existing file: 1$_: att3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1600.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1600.061 was not compressedSTATUS: $compression_test_returns existing file: 1$_: att4=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1605.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR_NRT.A2020065.1605.061 was not compressedSTATUS: $compression_test_returns existing file: 1$_: eph1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1550.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1550.061 was not compressedSTATUS: $compression_test_returns existing file: 1$_: eph2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1555.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1555.061 was not compressedSTATUS: $compression_test_returns existing file: 1$_: eph3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1600.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1600.061 was not compressedSTATUS: $compression_test_returns existing file: 1$_: eph4=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1605.061/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND_NRT.A2020065.1605.061 was not compressedSTATUS: $compression_test_returns existing file: 1$_: icefile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/063/N202006300_SEAICE_NSIDC_24h.hdf$_: met1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEP_6h.hdf$_: met2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEP_6h.hdf$_: met3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEP_6h.hdf$_: sstfile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/064/N2020064_SST_OIV2AV_24h.ncSTATUS: $ocssw_command_2 = modis_GEO.py --parfile=./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.GEO --threshold=95 --ancdb=./ancillary_data.dbSTATUS: $ocssw_return_2 = MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24scan: 0 out of 400 Thu Mar 5 15:03:16 2020...scan: 390 out of 400 Thu Mar 5 15:03:42 2020./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed{'geogen': {}}STATUS: The command: modis_GEO.py --parfile=./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_XvBvPIQs/MOD00.P2020065.1555.CNMI_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db RETURNED NO ERROR messageThis next bit is the days later reprocessing log snippet I just ran 2020-03-10 12:40:06:STATUS: $ocssw_command_1 = modis_L1A.py /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.PDS -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC --startnudge=0 --stopnudge=10[56208 refs]STATUS: $ocssw_return_1 = MODIS L1A version 6.0.6, built Aug 16 2019 12:46:26scan: 0 out of 404 Tue Mar 10 12:40:45 2020...scan: 390 out of 404 Tue Mar 10 12:40:47 2020$_: att1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1400.003/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1400.003 was not compressedSTATUS: $compression_test_returns existing file: 1$_: att2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1600.003/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1600.003 was not compressedSTATUS: $compression_test_returns existing file: 1$_: eph1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND.P2020065.1200.003/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND.P2020065.1200.003 was not compressedSTATUS: $compression_test_returns existing file: 1[66172 refs]$_: icefile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_SEAICE_NSIDC_24h.hdf$_: met1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEPR2_6h.hdf$_: met2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf$_: met3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf$_: ozone1=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_O3_AURAOMI_24h.hdf$_: ozone2=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf$_: ozone3=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf$_: sstfile=/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N2020065_SST_OIV2AV_24h.ncSTATUS: $ocssw_command_2 = modis_GEO.py --parfile=./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.CNMI_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db[66403 refs]STATUS: $ocssw_return_2 = MODIS GEO version 6.1.0, built Aug 16 2019 12:46:24scan: 0 out of 400 Tue Mar 10 12:40:49 2020...scan: 390 out of 400 Tue Mar 10 12:41:00 2020./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed{'geogen': {}}Percent valid data (34.00) is less than threshold (95.00)ERROR: MODIS geolocation processing failed.ERROR: The command: modis_GEO.py --parfile=./MOD00.P2020065.1555.CNMI_FULL.L1A_LAC.atteph.fixed /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.CNMI_FULL.L1A_LAC -o /work/b/bmurch/tmp/S4P_MODIS_H5_MODISA.CNMI.2020.065_version_0.10_lprskpTu/MOD00.P2020065.1555.CNMI_FULL.GEO --threshold=95 --ancdb=./ancillary_data.db returned an error code 'ERROR:' and likely 'ERROR: MODIS geolocation processing failed.'I have looked at the anc files and they appear correct:[bmurch@optics0 cron]$ file /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1400.003 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1600.003 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND.P2020065.1200.003 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_SEAICE_NSIDC_24h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEPR2_6h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_O3_AURAOMI_24h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N2020065_SST_OIV2AV_24h.nc/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1400.003: data/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1600.003: data/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND.P2020065.1200.003: data/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_SEAICE_NSIDC_24h.hdf: Hierarchical Data Format (version 4) data/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEPR2_6h.hdf: Hierarchical Data Format (version 4) data/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf: Hierarchical Data Format (version 4) data/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf: Hierarchical Data Format (version 4) data/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_O3_AURAOMI_24h.hdf: Hierarchical Data Format (version 4) data/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf: Hierarchical Data Format (version 4) data/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf: Hierarchical Data Format (version 4) data/shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N2020065_SST_OIV2AV_24h.nc: Hierarchical Data Format (version 5) datals -l returns-rw-rw-r-- 1 bmurch cms_optics 201285 Mar 7 06:56 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_O3_AURAOMI_24h.hdf-rw-rw-r-- 1 bmurch cms_optics 245446 Mar 6 18:12 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006500_SEAICE_NSIDC_24h.hdf-rw-rw-r-- 1 bmurch cms_optics 1650473 Mar 10 00:11 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006512_MET_NCEPR2_6h.hdf-rw-rw-r-- 1 bmurch cms_optics 1650461 Mar 10 00:11 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf-rw-rw-r-- 1 bmurch cms_optics 1650461 Mar 10 00:11 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N202006518_MET_NCEPR2_6h.hdf-rw-rw-r-- 1 bmurch cms_optics 1656193 Mar 6 07:51 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/N2020065_SST_OIV2AV_24h.nc-rw-rw-r-- 1 bmurch cms_optics 462848 Mar 10 00:14 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1400.003-rw-rw-r-- 1 bmurch cms_optics 462848 Mar 10 00:14 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1ATTNR.P2020065.1600.003-rw-rw-r-- 1 bmurch cms_optics 5531104 Mar 10 12:54 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/065/PM1EPHND.P2020065.1200.003-rw-rw-r-- 1 bmurch cms_optics 201285 Mar 8 07:55 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf-rw-rw-r-- 1 bmurch cms_optics 201285 Mar 8 07:55 /shares/cms_optics/apps/seadas/seadas-7.5.3/ocssw/var/anc/2020/066/N202006600_O3_AURAOMI_24h.hdf