seadas 7.4, update_luts.py

[lmajewsk]

Hi,
I think update_luts.py has regressed in the 7.4 release (lut_utils.py point to http rather than https).
After changing to https, leapsec and utcpole get updated.

However, LUTS fail to download:

$ update_luts.py -v terra
[ MODIS ]
+ leapsec.dat
+ utcpole.dat
[ MODIS: TERRA ]
Error downloading https://oceandata.sci.gsfc.nasa.gov/Ancillary/LUTs/modist/cal/OPER/
Error: /short/er8/packages/seadas/7.0/ocssw/run/var/modist/cal/OPER/index.html does not exist

I think it's to do with the OPER directory listing not landing in "index.html"... or something like that.
Cheers,
Leon

[gnwiii]

update_luts.py is fine in SeaDAS 7.4 here. Check that your scripts are really on branch v7.4:

$ cd $OCSSWROOT/run/scripts
$ git status
On branch v7.4
Your branch is up-to-date with 'origin/v7.4'.

nothing to commit, working tree clean


Problems like the one you found can occur when someone forgets to add  -b v7.4 to the iinstall_ocssw.py command options or when $OCSSWROOT was accidentally left pointing to an older install.

[eko1968]

Hi,
I am new in seadas-7.4 and just installed (re-installed) it yesterday several time as I am not able to update LUTs.
I know the problem of updating LUTs I am facing now has already been discussed in this OC forum.
I have tried to follow what suggested by peoples here, but I cannot resolve it.

My installation is as below:
Linux : Fedora26
SeaDAS installation dir: /home/eko/seadas-7.4

I used command like:
$ ./install_ocssw.py --install-dir=/home/eko/seadas-7.4/ocssw --git-branch=v7.4 --local=/tmp --aqua --seawifs
to install the scripts from /tmp to seadas-7.4/ocssw directory (not to home directory), and re-run installer by typing:
$ sudo ./install_ocssw.py --install-dir=/home/eko/seadas-7.4/ocssw --git-branch=v7.4 --aqua --seawifs

with the above steps, OCSSW menu on SeaDAS7.4 GUI has changed to "Update OC Processors"

Giving the command like:
$ update_luts.py --verbose aqua, or
$ /usr/bin/python $OCSSWROOT/run/scripts/update_luts.py --verbose aqua
only returned:
[ MODIS ]
Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later.

I also tried to use "pyverchk3.py" shared in this forum, but I cannot make it.
Maybe I did something wrong. The only thing I remember is I got the message of "No module named six.moves"
I am still working on it (pyverchk3.py).

Below is my git, phyton, java information:

$ cd $OCSSWROOT/run/scripts
$ git status
On branch v7.4
Your branch is up-to-date with 'origin/v7.4'.
nothing to commit, working tree clean

$ which python
/usr/bin/python
$ python --version
Python 2.7.13

$ which java
/usr/bin/java
$ java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

Do I have improper version of fedore, git, and java? or mistakenly install seadas or scripts on improper directory?

I deeply appreciate for any assistance, suggestion how to solve the problems.
So sorry for very very long post.

Many thanks,
Eko

[gnwiii]

It looks like the OCSSW python scripts should be using python2.  The "pyverchk3.py" is for configurations where the "python" command is version 3, which does not appear to be your situation.    It is worth noting that the Python community recommends that the "python" command should always be python2 , see PEP 394, and should not arise when using a linux distro's packages.

Your /usr/bin/python may lack the recent ciphers (usually provided by the openssl library) needed to connect to NASA's https servers.  This has been a problem with other distributions and also macOS, see Sean's post.  If you
do find that python2 in fedora 26 lacks the required ciphers, you can try anaconda2, which has worked with other distros.

I checked a vanilla fedora 25 server installation:

/usr/bin/openssl ciphers | tr ':' '\n' | egrep "ECDHE-ECDSA-CHACHA20-POLY1305|ECDHE-ECDSA-AES256-GCM-SHA384|ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-AES256-SHA384|ECDHE-ECDSA-AES128-SHA256"
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES256-SHA384
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES128-SHA256

If your system agrees with the above, Sean's suggestion of side-effects from sudo could well be your problem.   You can remove the non-working installation and install again without invoking  sudo, or you can create a temporary account for a "test" user and install SeaDAS 7 there.

[OB.DAACx - SeanBailey]

Fedora26 should have no issues with running our code or connecting to our servers (I've not tested it, but it is the latest version and should "just work").  You mention you installed with sudo...you should not' do that.  There is no reason to use sudo for installing the OCSSW code, and doing so may cause weird problems related to permissions.
Sean

[eko1968]

My fedora has the folowing:
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES256-SHA384
ECDHE-ECDSA-AES128-SHA256

Then the ciphers may not be the issue here, I guess.

So, I removed installed ocssw and tried to re-install processing program without sudo like:
$ ./install_ocssw.py --install-dir=$HOME/seadas-7.4/ocssw --git-branch=v7.4 --aqua --seawifs

But the problem still persists. Below I copied the last eight lines returned.
.
.
Cloning into '/home/eko/seadas-7.4/ocssw/run/scripts'...
Receiving objects: 100% (383/383), 203.98 KiB | 14.57 MiB/s, done.
Resolving deltas: 100% (264/264), done.
Python version acceptable
Installing seawifs-luts (13 of 15)
Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later.
Error - Could not install luts for seawifs

I also tried to use python2 rather than python like:
$ /usr/bin/python2 $OCSSWROOT/run/scripts/update_luts.py --verbose aqua
But the problem is still there.
Is my python not Anaconda python2?
As you suggested, I may need to use Anaconda python 2.
I will do that, and let me see the results.

Thanks.

[eko1968]

The problem is solved.
It is just fine after installing python Anaconda 2.

[gnwiii]

Glad it is working.  The OCSSW Processing System works fine on a very minimal fedora 25 server system here, so the failure will remain a puzzle.  Having said that, Fedora is not expected to be robust so oddities do happen.

[ana_dogliotti]

Hello,
I'm writing because I have the same error when trying to update the LUTs.

update_luts.py -v aqua
[ MODIS ]
Error! could not establish a network connection. Check your network connection.
If you do not find a problem, please try again later.

I've just installed locally SeaDAS v7.4. (Ubuntu 16.04.2 LTS) using:

./install_ocssw.py -i /home/usr/seadas-7.4/ocssw -b v7.4 -l /home/usr/Download --aqua

I checked the git, python and Java version and got this

git status
On branch master
Your branch is up-to-date with «origin/master».
nothing to commit, working directory clean

python --version
Python 2.7.12

java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntuxx.xxx.xx.xx-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

Following Sean's post I checked the openssl version but it seems to be OK

/usr/bin/python -c 'import ssl ; print ssl.OPENSSL_VERSION'
OpenSSL 1.0.2g  1 Mar 2016

I don't understand the "master" version of the git instead of "v7.4". Could this be the reason why it's not working the LUTs update?
Any help is highly appreciated!
Thanks

Ana

[gnwiii]

Ubuntu 16.04 LTS is a suitable platform for SeaDAS 7.4, so you are right to suspect the git results.  It is not unusual to encounter systems with multiple ocssw directories left from previous installs, e.g., $HOME/ocssw and /opt/ocssw as well as $HOME/seadas-7.4/ocssw.

To check the value of OCSWROOT, in a terminal run "$ echo OCSSWROOT=$OCSSWROOT" (the "$ " is the shell prompt string -- only type the stuff following the prompt).

In your case, you should get "OCSWROOT=/home/usr/seadas-7.4/ocssw".  

You didn't mention the directory were you ran "git status".   Since the problem seems to be the python scripts, check the "$OCSSWROOT/run/scripts" directory.  You should get:
$ cd $OCSSWROOT/run/scripts
$ git status
On branch v7.4
Your branch is up-to-date with 'origin/v7.4'.