Earthdata Forum

I run `install_ocssw.py` daily on a few machines to keep things up-to-date and have been logging errors like the following error on all nodes since 2019-03-27:

```
[root@userproc test]# ls /srv/imars-objects/west_fl_pen
ls: cannot access /srv/imars-objects/west_fl_pen: No such file or directory
[root@userproc test]# /opt/ocssw/install_ocssw.py -v             -i /opt/ocssw -g https://oceandata.sci.gsfc.nasa.gov/ocssw/ -a linux_64 --curl             --aqua --terra --olci

gitBase     = https://oceandata.sci.gsfc.nasa.gov/ocssw/
gitBranch   = v7.5
install dir = /opt/ocssw
arch        = linux_64

git user.name = "Default Seadas User"
git user.email = "seadas-user@localhost"
Installing bundles.sha256sum (1 of 13)
Downloading bundles.sha256sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3880  100  3880    0     0  12846      0 --:--:-- --:--:-- --:--:-- 12890
Loading checksum file.
Installing common (2 of 13)
Updating (fetch) existing repository -  /opt/ocssw/share/common
Switching to branch -  v7.5
Branch 'v7.5' set up to track remote branch 'v7.5' from 'origin'.
Reset branch 'v7.5'
Your branch is up to date with 'origin/v7.5'.
Pulling from remote repository
Already up to date.
Installing OCSSW_bash.env (3 of 13)
Downloading OCSSW_bash.env.v7.5
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3001  100  3001    0     0   9985      0 --:--:-- --:--:-- --:--:-- 10003
Installing ocrvc (4 of 13)
Updating (fetch) existing repository -  /opt/ocssw/share/ocrvc
Switching to branch -  v7.5
Branch 'v7.5' set up to track remote branch 'v7.5' from 'origin'.
Reset branch 'v7.5'
Your branch is up to date with 'origin/v7.5'.
Pulling from remote repository
Already up to date.
Installing modis (5 of 13)
Updating (fetch) existing repository -  /opt/ocssw/share/modis
Switching to branch -  v7.5
Branch 'v7.5' set up to track remote branch 'v7.5' from 'origin'.
Reset branch 'v7.5'
Your branch is up to date with 'origin/v7.5'.
Pulling from remote repository
Already up to date.
Installing modis/aqua (6 of 13)
Updating (fetch) existing repository -  /opt/ocssw/share/modis/aqua
Switching to branch -  v7.5
Branch 'v7.5' set up to track remote branch 'v7.5' from 'origin'.
Reset branch 'v7.5'
Your branch is up to date with 'origin/v7.5'.
Pulling from remote repository
Already up to date.
Installing modis/terra (7 of 13)
Updating (fetch) existing repository -  /opt/ocssw/share/modis/terra
fatal: unable to access 'https://oceandata.sci.gsfc.nasa.gov/ocssw/modisterra.git/': Operation timed out after 300009 milliseconds with 0 out of 0 bytes received
Error - Could not run "cd /opt/ocssw/share/modis/terra; git fetch"
```

Note that running the command in the error seems to work just fine:

```
[root@userproc terra]# cd /opt/ocssw/share/modis/terra; git fetch
[root@userproc terra]# git status
On branch v7.5
Your branch is up to date with 'origin/v7.5'.
nothing to commit, working tree clean
```

Is `install_ocssw.py` working for anyone else?

PS: environment info:
```
[root@userproc terra]# cat /etc/*elease | head -n1
CentOS Linux release 7.6.1810 (Core)
[root@userproc terra]# python --version
Python 2.7.5
```

First, you really don't need to update daily.  Trust me, we aren't changing things that fast :wink:
You could run update_luts.py for the aqua and terra sensors monthly to keep the calibration updates current (or if you're really keen, weekly), but the code can be checked on a much lower frequency.

That said, the error looks like you may be tripping a firewall limit.  I just ran your command successfully on my laptop from the comfort of my couch, so it's not install_ocssw.py.
You mention, "on all nodes".  If all those nodes are coming into our servers from the same IP, and all are running the script at the same time you may want to try cycling
through one at a time.

Sean

I often encounter internet "glitches" running install_ocssw.py, so I expect to run the script multiple times.  It might be better to run a separate update for each sensor, or you can omit the sensors that previously succeeded in subsequent attempts.   If the script fails in the final stages updating LUT's you can run the update scripts rather than going thru the installer.   All this is a bit much to automate.  

I tried a version of your command adapted for my configuration on Ubuntu 16.04.  There were some problems.  The first was recovered, the second resulted in an
incomplete update:

install_ocssw.py -v -i $HOME/ocssw -g https://oceandata.sci.gsfc.nasa.gov/ocssw/ -a linux_64 --curl --aqua --terra --olci
[...]
Installing modis/terra (7 of 13)
Downloading new directory -  /home/seadas/ocssw/share/modis/terra
Downloading modisterra.bundle
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:05:00 --:--:--     0
Warning: Transient problem: timeout Will retry in 5 seconds. 5 retries left.
100  288M  100  288M    0     0  53.2M      0  0:00:05  0:00:05 --:--:-- 61.9M
comparing checksum for modisterra.bundle
Cloning into '/home/seadas/ocssw/share/modis/terra'...
Receiving objects: 100% (177/177), 288.39 MiB | 59.85 MiB/s, done.
Resolving deltas: 100% (34/34), done.
Checking connectivity... done.
Checking out files: 100% (174/174), done.
Updating (fetch) existing repository - /home/seadas/ocssw/share/modis/terra
From https://oceandata.sci.gsfc.nasa.gov/ocssw/modisterra
   a4f1114..db7095e  master     -> origin/master
   a4f1114..db7095e  v7.5       -> origin/v7.5
Switching to branch -  v7.5
Branch v7.5 set up to track remote branch v7.5 from origin.
Switched to a new branch 'v7.5'
Pulling from remote repository - /home/seadas/ocssw/share/modis/terra
[...]
Installing scripts (11 of 13)
Updating (fetch) existing repository -  /home/seadas/ocssw/scripts
From https://oceandata.sci.gsfc.nasa.gov/ocssw/scripts
   739aab2..0ec4351  v7.5       -> origin/v7.5
   739aab2..0ec4351  master     -> origin/master
Switching to branch -  v7.5
Branch v7.5 set up to track remote branch v7.5 from origin.
Reset branch 'v7.5'
Your branch is up-to-date with 'origin/v7.5'.
Pulling from remote repository
fatal: unable to access 'https://oceandata.sci.gsfc.nasa.gov/ocssw/scripts.git/': Operation timed out after 0 milliseconds with 0 out of 0 bytes received
Error - Could not run "cd /home/seadas/ocssw/scripts; git pull --progress"


I notice you installed the OCSSW software as "root".   General "best practices" minimize the use of elevated privileges, and newer systems may impose limits on
what can be done with "root" privileges (e.g., no GUI's) so can produce weird errors.   If you wish to install to /opt/ocssw you can create the directory using root privileges and change ownership to the "seadas" user, then run the installation as the "seadas" user.

You are definitely right about install as "root" being in general a bad idea, but it was the quickest hack that got things working on these limited-use systems.

I had only minimal issues with `install_ocssw.py` previous to the aforementioned date and the error now seems quite consistent.
Your suggestions are good, but perhaps this automated retry logic should be in the script itself?
I don't really have time to debug this right now so in the meantime I have temporarily disabled updates.
If anyone has suggestions or a fix is deployed let me know and I will flip the switch and test it.

Oops, just saw your response about the firewall.
I suspect updating too often and being blocked by a firewall is my issue.
I will consider deploying a cronjob instead of using puppet when I have time to revisit this.

Thanks!