ocssw 7.5 modis_L1A.py failed to process some pds

[pavel_babyak]

Hello.

After a recent update, some MODIS files are no longer processed.

To check error i have decided to install the ocssw on a clean machine with the following script:

#!/bin/bash
export OCSSWROOT=$HOME/ocssw
mkdir -p ${OCSSWROOT}

BDIR=${HOME}/.local/bin
wget -O ${BDIR}/install_ocssw.py https://oceandata.sci.gsfc.nasa.gov/ocssw/install_ocssw.py || exit
chmod +x ${BDIR}/install_ocssw.py
${BDIR}/install_ocssw.py -i ${OCSSWROOT} --git-branch=v7.5 -a linux_64 --avhrr --aqua --terra


For example, i took two files from 2019/09/03 and 2007/04/05
Second file was handled correctly by previous versions.
Data files and log files you can look at ftp://ftp.satellite.dvo.ru/pub/ocssw/

That's what gave ocssw on the first file.
pdsinfo output:

$ ocssw pdsinfo P1540064AAAAAAAAAAAAAA19246030843001.PDS
pdsinfo V1.6 (Aug 16 2019)
APID 64: count 444011 invalid 0 missing 18
first packet: 2019/09/03 03:08:42.769570
last packet: 2019/09/03 03:12:18.427976
missing seconds: 0
day packets: 396707/46720
night packets: 0/0
engineering packets: 584/0


modis_L1A.py output:

$ ocssw modis_L1A.py --verbose --log --mission aqua --output P1540064AAAAAAAAAAAAAA19246030843001.L1A P1540064AAAAAAAAAAAAAA19246030843001.PDS
Determining pass start and stop time...

Using specified output L1A filename: P1540064AAAAAAAAAAAAAA19246030843001.L1A
Input Level 0: P1540064AAAAAAAAAAAAAA19246030843001.PDS
Output Level 1A: P1540064AAAAAAAAAAAAAA19246030843001.L1A
Satellite: aqua
Start Time: 2019-09-03T03:10:00.000000Z
Stop Time: 2019-09-03T03:10:00.000000Z
Granule Duration: 0.0 seconds

Processing MODIS L0 file to L1A...
MODIS L1A version 6.0.6, built Aug 16 2019 12:46:26
l1agen_modis exit status: 0
MODIS L1A processing complete.


As you can see, pdsinfo returns the correct time of the first and last packet.
But in the modis_L1A.py "Start Time" and "Stop Time" are equals 2019-09-03T03:10:00.000000Z
However it ended without error, return code is 0 and no L1A-file generated.

Forth on the second file.
pdsinfo output:

$ ocssw pdsinfo PM0704050335.pds
pdsinfo V1.6 (Aug 16 2019)
APID 64: count 108151 invalid 0 missing 12215
first packet: 2007/04/05 03:41:20.519309
last packet: 2007/04/05 03:42:18.126695
missing seconds: 4
day packets: 96811/11200
night packets: 0/0
engineering packets: 140/0


modis_L1A.py output:

$ ocssw modis_L1A.py --verbose --log --mission aqua --output PM0704050335.L1A PM0704050335.pds
Determining pass start and stop time...

Using specified output L1A filename: PM0704050335.L1A
Input Level 0: PM0704050335.pds
Output Level 1A: PM0704050335.L1A
Satellite: aqua
Start Time: 2007-04-05T03:40:00.000000Z
Stop Time: 2007-04-05T03:40:00.000000Z
Granule Duration: 0.0 seconds

Processing MODIS L0 file to L1A...
MODIS L1A version 6.0.6, built Aug 16 2019 12:46:26
l1agen_modis exit status: 1
modis_l1a: ERROR: MODIS L1A processing failed.
Please examine the LogStatus and LogUser files for more information.


In this case modis_L1A.py also misread the time, but now returned with an error.

We have noticed that not all files are handled with errors. But only those that are smaller than some threshold.

Has anyone met a similar problem?
Any workaround?

PS ${HOME}/.local/bin/ocssw:
----------------------
#!/bin/bash

export LANG=C
export OCSSWROOT=${HOME}/ocssw
source ${OCSSWROOT}/OCSSW_bash.env
ulimit -s unlimited
"${@}"
----------------------

[OB.DAACx - SeanBailey]

I am unable to connect to your FTP server.  Please attach you logs or post a snippet of any errors you see in them.
BTW, which previous version worked for you (the MODIS L1/GEO code itself hasn't changed in quite sometime, although we have recompiled it).

Sean

[pavel_babyak]

Sorry, we had some technical issues with our servers, now all should works fine.
If not you can get logs from dropbox - https://www.dropbox.com/s/er1ps3cbu7wxjg1/ocssw.out.tgz

As for which version worked correctly, i'm not exactly sure, because the previous update was made in the first quarter of 2019, but we are working with 7.5 brunch.

[gfireman]

Pavel -

You have found a bug!  In code that I wrote, even.

In operational processing, we need to make sure the last scan is processed even if it's partial, so we round the granule endtimes to the nearest 5-min boundary.  It turns out that approach doesn't always work - in your examples, both endtimes were closest to the same time.  We're working on a solution.

In the meantime, the workaround is to comment out the two lines that perform the rounding.
In $OCSSWROOT/scripts/modules/modis_L1A_utils.py:
        #self.start = ProcUtils.round_minutes(self.start,5,'t')
        #self.stop  = ProcUtils.round_minutes(self.stop, 5,'t')

- Gwyn