confusing getanc.py behaviour

bruce · by **bruce** » Mon Aug 28, 2017 9:27 am America/New_York

The actions of getanc.py don't always behave as one might expect. See for example...

[bcb@modis allpic]$ getanc.py A2017184170500.L1A_LAC
*** WARNING: No optimal MET files found.
*** WARNING: No optimal MET files found.
*** WARNING: No optimal SST files found.
*** WARNING: No optimal ICE files found.

*** WARNING: The following ancillary data types were missing or are not optimal: MET OZONE SST Sea Ice
[bcb@modis allpic]$ getanc.py A2017184170500.L1B_LAC
Aqua
icefile=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718400_SEAICE_NSIDC_24h.hdf
met1=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718412_MET_NCEPR2_6h.hdf
met2=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
met3=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
ozone1=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718400_O3_AURAOMI_24h.hdf
ozone2=/seadas/seadas-7.4/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
ozone3=/seadas/seadas-7.4/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
sstfile=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N2017184_SST_OIV2AV_24h.nc
[bcb@modis allpic]$

The only (apparent) difference between the 2 calls is one has an A as the 18th character of the filename, while the other has a B (yes, I know there are internal differences in the file :-)

Why does the first fail?

by **OB SeaDAS - dshea** » Mon Aug 28, 2017 10:23 am America/New_York

Unfortunately, getanc.py works fine for me on that L1A MODIS file. Try the verbose option to see if the script gets the date correct.

getanc.py -v A2017184170500.L1A_LAC

don

bruce · by **bruce** » Mon Aug 28, 2017 10:43 am America/New_York

(Of course it works for you, I'd expect nothing less... Ask Sean how many times it fails for me and works for him :-)

[bcb@modis allpic]$ getanc.py -v A2017184170500.L1A_LAC
Searching database: /seadas/seadas-7.4/ocssw/run/var/ancillary_data.db

Input file: A2017184170500.L1A_LAC
Start time: 2017184170500
End time: 2017184170959

*** WARNING: No optimal MET files found.
*** WARNING: No optimal MET files found.
*** WARNING: No optimal SST files found.
*** WARNING: No optimal ICE files found.

Created 'A2017184170500.L1A_LAC.anc' l2gen parameter text file:

*** WARNING: The following ancillary data types were missing or are not optimal: MET OZONE SST Sea Ice
[bcb@modis allpic]$ getanc.py -v A2017184170500.L1B_LAC
Searching database: /seadas/seadas-7.4/ocssw/run/var/ancillary_data.db

Input file: A2017184170500.L1B_LAC
Start time: 2017184170500
End time: 2017184170959

Found: /seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
Found: /seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
Found: /seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718412_MET_NCEPR2_6h.hdf
Found: /seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718400_O3_AURAOMI_24h.hdf
Found: /seadas/seadas-7.4/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
Found: /seadas/seadas-7.4/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
Found: /seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N2017184_SST_OIV2AV_24h.nc
Found: /seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718400_SEAICE_NSIDC_24h.hdf

Created 'A2017184170500.L1B_LAC.anc' l2gen parameter text file:

icefile=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718400_SEAICE_NSIDC_24h.hdf
met1=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718412_MET_NCEPR2_6h.hdf
met2=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
met3=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
ozone1=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N201718400_O3_AURAOMI_24h.hdf
ozone2=/seadas/seadas-7.4/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
ozone3=/seadas/seadas-7.4/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
sstfile=/seadas/seadas-7.4/ocssw/run/var/anc/2017/184/N2017184_SST_OIV2AV_24h.nc

- All optimal ancillary data files were determined and downloaded. -
[bcb@modis allpic]$

bruce · by **bruce** » Mon Aug 28, 2017 10:48 am America/New_York

Not shown here, but it's worth noting that in both cases, 'echo $?' prints 0.

by **OB SeaDAS - dshea** » Mon Aug 28, 2017 10:55 am America/New_York

Strange. Once the script gets the start time, it should not care what kind of file you used as input. I'll take a look at the script and see if I see anything that could cause this.

don

gnwiii · by **gnwiii** » Mon Aug 28, 2017 1:23 pm America/New_York

Before going further, we should verify that your file is the same as one that works:

$ shasum A2017184170500.L1A_LAC
a84dc3904273d412380eb4d478099bfcc058e3e6  A2017184170500.L1A_LAC

Once you have run getanc.py, it seems to check the database before it goes to "Determining pass start and end times". The file that caused the problem worked here (Bedford Inst. in Canada) on July 4th (while the US was partying and not bogging down the internet). Using a different account the initial run gives:

$ getanc.py -v A2017184170500.L1A_LAC
Searching database: /home/seadas2/ocssw/run/var/ancillary_data.db 
Determining pass start and end times...
Aqua

Input file: A2017184170500.L1A_LAC
Start time: 2017184170500
End time: 2017184170959

Downloading 'N201718418_MET_NCEPR2_6h.hdf' to /home/seadas2/ocssw/run/var/anc/2017/184
  Found: /home/seadas2/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
Downloading 'N201718412_MET_NCEPR2_6h.hdf' to /home/seadas2/ocssw/run/var/anc/2017/184
Downloading 'N201718400_O3_AURAOMI_24h.hdf' to /home/seadas2/ocssw/run/var/anc/2017/184
Downloading 'N201718500_O3_AURAOMI_24h.hdf' to /home/seadas2/ocssw/run/var/anc/2017/185
  Found: /home/seadas2/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
Downloading 'N2017184_SST_OIV2AV_24h.nc' to /home/seadas2/ocssw/run/var/anc/2017/184
Downloading 'N201718400_SEAICE_NSIDC_24h.hdf' to /home/seadas2/ocssw/run/var/anc/2017/184

Created 'A2017184170500.L1A_LAC.anc' l2gen parameter text file:

icefile=/home/seadas2/ocssw/run/var/anc/2017/184/N201718400_SEAICE_NSIDC_24h.hdf
met1=/home/seadas2/ocssw/run/var/anc/2017/184/N201718412_MET_NCEPR2_6h.hdf
met2=/home/seadas2/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
met3=/home/seadas2/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
ozone1=/home/seadas2/ocssw/run/var/anc/2017/184/N201718400_O3_AURAOMI_24h.hdf
ozone2=/home/seadas2/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
ozone3=/home/seadas2/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
sstfile=/home/seadas2/ocssw/run/var/anc/2017/184/N2017184_SST_OIV2AV_24h.nc

- All optimal ancillary data files were determined and downloaded. -

The output is different on subsequent runs:

$ getanc.py -v A2017184170500.L1A_LAC --timeout 0.1
Searching database: /home/seadas2/ocssw/run/var/ancillary_data.db 

Input file: A2017184170500.L1A_LAC
Start time: 2017184170500
End time: 2017184170959

  Found: /home/seadas2/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
  Found: /home/seadas2/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
  Found: /home/seadas2/ocssw/run/var/anc/2017/184/N201718412_MET_NCEPR2_6h.hdf
  Found: /home/seadas2/ocssw/run/var/anc/2017/184/N201718400_O3_AURAOMI_24h.hdf
  Found: /home/seadas2/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
  Found: /home/seadas2/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
  Found: /home/seadas2/ocssw/run/var/anc/2017/184/N2017184_SST_OIV2AV_24h.nc
  Found: /home/seadas2/ocssw/run/var/anc/2017/184/N201718400_SEAICE_NSIDC_24h.hdf

Created 'A2017184170500.L1A_LAC.anc' l2gen parameter text file:

icefile=/home/seadas2/ocssw/run/var/anc/2017/184/N201718400_SEAICE_NSIDC_24h.hdf
met1=/home/seadas2/ocssw/run/var/anc/2017/184/N201718412_MET_NCEPR2_6h.hdf
met2=/home/seadas2/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
met3=/home/seadas2/ocssw/run/var/anc/2017/184/N201718418_MET_NCEPR2_6h.hdf
ozone1=/home/seadas2/ocssw/run/var/anc/2017/184/N201718400_O3_AURAOMI_24h.hdf
ozone2=/home/seadas2/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
ozone3=/home/seadas2/ocssw/run/var/anc/2017/185/N201718500_O3_AURAOMI_24h.hdf
sstfile=/home/seadas2/ocssw/run/var/anc/2017/184/N2017184_SST_OIV2AV_24h.nc

- All optimal ancillary data files were determined and downloaded. -

What happens if you try running getanc.py on a different account (or after temporarily renaming the database and run/var/anc/2017/184/ directory)?

bruce · by **bruce** » Mon Aug 28, 2017 2:39 pm America/New_York

No "plain old" shasum on this box, but

[bcb@modis allpic]$ sha256sum A2017184170500.L1A_LAC
bae41b5209f94100e3c54ee4f728877b3948f60b7df0d8d3f3f3b7a3b37eb1b3 A2017184170500.L1A_LAC
[bcb@modis allpic]$ sha1sum A2017184170500.L1A_LAC
692264116b3d4af117963d52dcb26774e4df4ba3 A2017184170500.L1A_LAC

semi-interstingly, add --refreshDB and it works. Guess I'll make that the default from now on :-)

OB.DAACx - SeanBailey · by **OB.DAACx - SeanBailey** » Mon Aug 28, 2017 3:04 pm America/New_York

Couple of things to note...

1) The shasum (or sha1sum) for a MODIS L1A file *can* differ but the files be the 'same'. The geolocation code will alter the metadata of the L1A file, so if you've run that on the file, it will not have the same checksum, but unless the process borked, the file will have the same data.

2) You should probably only use --refreshDB if there is a problem. If the local DB is happy and the file is fine, running with refreshDB will unnecessarily contact our servers (not that it's a problem for us, but may slow you down)

Sean

bruce · by **bruce** » Mon Aug 28, 2017 4:14 pm America/New_York

I find this "behavior" occurring between 30 and 50% of the time. I'll stick with --refreshDB to avoid having to process twice to get the best results...

gnwiii · by **gnwiii** » Mon Aug 28, 2017 6:48 pm America/New_York

I had forgotten that geolocation alters L1A files. It would be useful to know if the altered files are reproducible. The majority of issues I have seen for getanc.py were either network issues or conflicts when multiple getanc.py were run (e.g., a batch process appeared to hang so the script was run a second time without killing the original process). You could try running with verbose and different timeout values to see if there is a pattern to the failures.