l2gen MOD02/MYD02

[cassandranormandin]

Ok, thank you for your reply.
I will work with the command-line now.
I've downloaded my files on ladsweb and for the same date and hour, MOD021KM and MOD02HKM don't have the same name. So, it doesn't work. Where did you download your files ?
Example for A2013191:
                - MOD021KM.A2013191.1015.006.2014220154608.hdf
                - MOD02HKM.A2013191.1015.006.2014219112150.hdf

Thank you again,
Cassandra

[gfireman]

Cassandra -

Files distributed by LAADS have the processing timestamp as part of the filename.  Usually the three L1B resolutions are created at exactly the same time, so our code looks for higher-resolution files with the same suffix.  One workaround would be to remove the processing timestamp from your input files.  You could do this in the bash command line:

for f in M[OY]D02[1HQ]KM.A20[0-9]*.[0-9]*.006.[0-9]*.hdf
  do mv $f `echo $f | cut -d\. -f1-4,6`
done

A better option would be to create the L1B files yourself.  The LAADS L1B calibration coefficients are tuned for high-radiance land scenes.  The L1B calibration coefficients we provide have been optimized for ocean color applications.  Furthermore, the cross-calibration coefficients applied in l2gen are specifically tuned to our set of L1B coeffs, and should not be applied to LAADS L1B files.

If you haven't done any spatial extraction, it makes no difference if you download L1A and GEO files from us or from LAADS.

[gnwiii]

The files I used were also from ladsweb.  Some groups of 1KM, HKM, and QKM files differ by a small time interval in "production date and time". It seems that l2gen wants the "production date and time" of the 1KM, HKM, and QKM files to match.  In a terminal you should see a message like:
-W- /data5/swdev/seadas/build/src/l2gen/l1_hmodis_hdf.c:217: Cannot open file: ./MOD02HKM.A2013191.1015.006.2014220154608.hdf
File not found: ./MOD02HKM.A2013191.1015.006.2014220154608.hdf
Processing at 500m resolution requires a HKM file to be present in the same directory as the 1KM L1B file.

Did the files that worked all share the same "production date and time"?  It can be helpful to preserve the original file name, so you might want to create symbolic links for the HKM (and QKM if required) files to names that have the same "production date and time" as the 1KM file, e.g.,

$ ln -s MOD02HKM.A2013191.1015.006.2014219112150.hdf MOD02HKM.A2013191.1015.006.2014220154608.hdf


[cassandranormandin]

Thank you gfireman for the informations.
Thank you gnwii too!
Yes, I've seen this message in my terminal when the names are not the same. After changing the name of my HKM file, it works well using command line!
I've exactly the same l2gen.log as your file.

In the SeaDAS GUI, files which worked last week have not the same name... I don't understand but I am working with command line, no problem.

Thank you,
Cassandra

[gnwiii]

Maybe the GUI has some logic deal with small differences in the "processing timestamp".   Does the GUI work when the processing timestamp is the same or has been removed from the filenames?

The GUI is handy when you are working out options for some one-off processing, but for routine processing the command-line has advantages.  Linux is based on Unix, which was invented at a time when graphical displays were rare and expensive.  A lot of thought went into making command-line processing efficient so if you are doing a lot of processing it will pay off to invest some time learning more about the command-line.  There are good resources, such as Linux Command.

For the benefit of future readers who encounter this thread, I should mention that the simple examples discussed in this thread omit some important details such as downloading and using ancillary files.

[cassandranormandin]

No, the GUI doesn't work when the timestamp is the same/changed.
Thank you again,
Cassandra

[gnwiii]

Totally understandable if you don't have time to investigate the processing failures with the GUI, but it could be helpful to others if we can determine why the GUI is failing.

Do you start the GUI from a terminal or using a menu or icon?  Starting the GUI (run "~/seadas-7.4/bin/seadas") in a terminal where l2gen is working should use the SeaDAS-7.4+OCSSW settings that work for command-line l2gen.

Differences between the GUI and the command-line can occur if there is some mismatch between the directories used for programs and data files.  This is a common problem for people who have used multiple SeaDAS versions over a period of years, and may have multiple OCSSW directories from old installs and old configuration settings of environment variables in the bash startup files ($HOME/.bashrc and $HOME/.profile or $HOME/.bash_profile, or the system-wide startup files in the /etc directory.  The GUI sets the OCSSW ___location at the end of the file <seadas_install_dir>/config/seadas.config.  A typical installation for user "sdsuser" would have:

$ tail -5 ~sdsuser/seadas-7.4/config/seadas.config

  # set ___location of OCSSWROOT
  seadas.ocssw.root = ${seadas.home}/ocssw
  seadas.ocssw.___location = local


On a fresh install, the last 5 lines of seadas.env would be identical to the above.

[cassandranormandin]

No problem, I have time to try to understand too. I would like to know why it doesn't work too.
I always start SeaDAS with a terminal. I have the same 5 lines in my terminal!

I was trying to update OC processors and this is the message I had (maybe link with l2gen fail?):
execution exception: java.io.IOException: install_ocssw.py failed with exit code 1.
Check log for more details.
  File "/home/cnormandin/seadas-7.4/ocssw/run/scripts/install_ocssw.py", line 51
    print 'Loading checksum file.'
                                 ^
SyntaxError: Missing parentheses in call to 'print'

To finish, I don't understand why it works well in command line for my first image (MOD021KM.A201319 at 1 km with geofile (MOD03), MOD02HKM and MOD02QKM) but for an other image it doesn't work (I try to do the processing on several images). For example for the image MOD021KM.A2017170.1015.006.2017170192412.hdf, I have MOD03, MOD02HKM and MOD02QKM associated (MOD03.A2017170.1015.006.2017170170944.hdf, MOD02QKM.A2017170.1015.006.2017170192412.hdf and MOD02HKM.A2017170.1015.006.2017170192412.hdf). The four files are in the same folder. This is the error message:
-W- /data5/swdev/seadas/build/src/l2gen/l1_hmodis_hdf.c:217: Cannot open file: /home/cnormandin/Terra/MOD02HKM.A2017170.1015.006.2017170192412.hdf
File not found: /home/cnormandin/Terra/MOD02HKM.A2017170.1015.006.2017170192412.hdf
Processing at 250m resolution requires a HKM file to be present in the same directory as the 1KM L1B file.

This is the processing I made on different images (that works well for the first image!):
l2gen ifile=/home/cnormandin/Terra/MOD021KM.A2017170.1015.006.2017170192412.hdf geofile=/home/cnormandin/Terra/MOD03.A2017170.1015.006.2017170170944.hdf ofile=/home/cnormandin/Corrections_atmospheriques/T2017170.L2_LAC_OC suite=OC l2prod=Rrs_nnn aer_opt=-1 maskhilt=0 maskland=0 north=45.6 south=44.6 east=0.1 west=-1.2 cloud_thresh=0.018 cloud_wave=2130 resolution=250
But the files are in the same directory....

thank you in advance,
Cassandra

[OB SeaDAS - dshea]

Your first problem is that the default python is version 3 and not version 2.  install_ocssw.py is a python 2.7 program.

Not sure why l2gen is not working for you.  I downloaded these files:

dshea@gs616-crab:~/data/terra$ ls -l
total 522375
-rw-rw-r-- 1 dshea dshea 161605822 Jul 13 09:06 MOD021KM.A2017170.1015.006.2017170192412.hdf
-rw-rw-r-- 1 dshea dshea 145904465 Jul 13 09:06 MOD02HKM.A2017170.1015.006.2017170192412.hdf
-rw-rw-r-- 1 dshea dshea 155513018 Jul 13 09:06 MOD02QKM.A2017170.1015.006.2017170192412.hdf
-rw-rw-r-- 1 dshea dshea  38051813 Jul 13 09:06 MOD03.A2017170.1015.006.2017170170944.hdf

l2gen works fine at resolution=250 on the command line and in the GUI.  The GUI might have a problem if you have python3 as you default python, but the command line l2gen does not use python.  Here is the command line that works for me:

dshea@gs616-crab:~/data/terra$ l2gen ifile=MOD021KM.A2017170.1015.006.2017170192412.hdf geofile=MOD03.A2017170.1015.006.2017170170944.hdf ofile=junk.L2.nc resolution=250

[gnwiii]

The message:

  File "/home/cnormandin/seadas-7.4/ocssw/run/scripts/install_ocssw.py", line 51
    print 'Loading checksum file.'
                                 ^
SyntaxError: Missing parentheses in call to 'print'
indicates that your python program is the "new and improved" python3.  In a terminal, you probably see something like:
$ python -c 'import sys; print(sys.version)'
Python 3.4.3 (default, Nov 17 2016, 01:08:31)
[GCC 4.8.4]


In that case, you probably also have something like:
$ python2 -c 'import sys; print(sys.version)'
2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4]


The OCSSW processing system requires python2.7 or later (but on some linux distros, the https support library is outdated and doesn't work with NASA's https servers).  Many Linux distros (as well as MacOS) use python for system management, and newer distro releases are moving to making python3 the default.   The OCSSW processing system includes a script, pyverchk.py, that checks for really old python2 versions, allows you to enter the full path of a suitable version (e.g., /usr/bin/python2.7).  Many people use packages that assume python3 is the default, so in our lab I modified pyverchk.py so it can be run using python2 or python3.   The new script is called pyverchk3.py (search the forum for "pyverchk3").  It should be kept separate from the OCSSW files so it doesn't get removed when you update the OCSSW processing system.