Create a composite image from 5 data

[antu]

Hello,
We are working with monthly level-3 SMI chl-a data from 2012-2017. We want to create 5-year composite image for each month. For example:one image January month will be a composite image of all January month images from 2012 to 2017.
What tools can we use to do this? And How can I do this?
We are using SeaDAS 7.5.3 in windows 10.
Regards,
Antu

[OB.DAACx - SeanBailey]

The easiest way to do this is with l3bin.  Make sure you have the L3b files for the data you wish to composite, then pass the appropriate monthly files as input to l3bin to create a climatology.You can then map the resulting bin file climatologies using l3mpagen.

On Windows 10, you'll need to use the client/server approach (using WSL works nicely for this). 

Sean

[antu]

Thank SeanBailey for your such helpful suggestions.
As you suggested, I am moving with client/server approach using WSL. I have already installed ubuntu WSL, oracle java8 , python2.7 and git as instructed in https://seadas.gsfc.nasa.gov/client_server/. But from 3. Deploy the OCSSW Web Services Package, I dont get how to move. I am just stuck.

My questions:
I have downloaded seadas-ocsswserver.jar and ocsswserver.config files and as said at that instruction "For our example we will put them in the home directory", so, I put them into my wsl root "C:\Users<username>\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState/rootfs/home/<username>" But those two files wont show up into wsl bash. So, I have created a shareable folder folder into C drive using "alias" command and copy those two files in that "shareable folder" and after typing "ls" command those two files showed up in bash.
Question 1: Is it necessary to move those two files in home directory ("C:\Users<username>\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79rhkp1fndgsc\LocalState/rootfs/home/<username>") of WSL ? If it is necessary, after copying them into home folder, how can I show up those files in bash?

Question 2: What will be my task with this two files- install or run the jar or config file?

Question 3: Where I will put that configuration (below), which should I modified?

Example ocsswserver.config for WSL:
ocsswrest.version=1.0
baseUriPortNumber=6400
ocsswroot=${user.home}/ocssw
serverWorkingDirectory=/mnt/c/Users/${user.name}/seadasClientServerShared
clientServerSharedDir=true
keepIntermediateFilesOnServer=false
processInputStreamPortNumber=6402
processErrorStreamPortNumber=6403

Question 3: For "4. SeaDAS Client Configuration",  How to modify (c:/program/seadas/config/seadas.config) -this file with this below configuration, as windows seadas.config has a configuration about  10 lines long text, which part I should modify with this below code?
Example for WSL:
   seadas.ocssw.___location=virtualMachine
   seadas.ocssw.port=6400
   seadas.ocssw.sharedDir=C:\\Users\\${user.name}\\seadasClientServerShared
   seadas.client.id=${user.name}
   seadas.ocssw.keepFilesOnServer=false
   seadas.ocssw.processInputStreamPort=6402
   seadas.ocssw.processErrorStreamPort=6403

Question 4: I have started server with this "$ java -Xmx4G -jar seadas-ocsswserver.jar ocsswserver.config" command and it started as I found this (Press 'Ctrl' + 'C'  to stop it...) output at the end of bash. But when I open seadas in windows and clicking "OCSSW" is says "Remote server is down. OCSSW is not accessible. Please start OCSSW remote server". 
I have replaced the last time (one of these three values: "local", "virtualMachine", # or the remote server address. #seadas.ocssw.___location= virtualMachine...............processErrorStreamPort=6403) of "seadas.config" file (c:/program/seadas/config/seadas.config), with this code "seadas.ocssw.___location=virtualMachine.seadas.ocssw.port=6400.seadas.ocssw.sharedDir=${user.dir}/seadasClientServerShared.seadas.client.id=${user.name}.seadas.ocssw.keepFilesOnServer=false.seadas.ocssw.processInputStreamPort=6402.seadas.ocssw.processErrorStreamPort=6403"

I think, you have got my points. If there are any query, please let me know.
Thank you,
Antu

[OB.DAACx - SeanBailey]

> Question 1: Is it necessary to move those two files in home directory ?

They need to reside in  (or at least be accessible from) the WSL environment.  When you launch WSL you're given a bash shell with the current working directory as the WSL home directory.

> Question 2: What will be my task with this two files- install or run the jar or config file?

First, create the configuration file (e.g. ocsswserver.config, but you can call it whatever you like)
Second, (once the prerequisites are installed and the seadas-ocsswserver.jar is in place), run: java -Xmx4G -jar seadas-ocsswserver.jar ocsswserver.config to launch the server.

> Question 3: Where I will put that configuration (below), which should I modified?

It can be anywhere.  We just suggest you put it in the WSL home directory as that's where you'll be when you launch any new WSL shell.
In the server config, you should replace the ${user.name} and ${user.home} with the explicit values rather than the placeholders.

> Question 3: For "4. SeaDAS Client Configuration",  How to modify (c:/program/seadas/config/seadas.config) -this file with this below configuration, as windows seadas.config has a configuration about  10 lines long text, which part I should modify with this below code?

Just append it to the bottom of the seadas.config file

Sean

[antu]

Thanks Sir,
I think, it will work. Though, I am still facing some problems.

As you said to replace the ${user.name} and ${user.home} with the explicit values in the server config. Here,  I did it as

ocsswroot=${user.home}/ocssw
${user.name}= /mnt/c/Users/antu/seadasClientServerShared/ocssw
serverWorkingDirectory=/mnt/c/Users/antu/seadasClientServerShared

Then,I have uncommented out the following in your seadas.config file:

# settings used for the remote OCSSW server
#
# the seadas.ocssw.___location property should be assigned one of these three values: "local", "virtualMachine",
# or the remote server address.
seadas.ocssw.___location =virtualMachine
seadas.ocssw.port=6400
seadas.ocssw.sharedDir=C:\\Users\\${antu}\\seadasClientServerShared
seadas.client.id=${antu}
seadas.ocssw.keepFilesOnServer=false
seadas.ocssw.processInputStreamPort=6402
seadas.ocssw.processErrorStreamPort=6403

Quesetion 1: Did I explicit the correct value of server config and did I correctly uncomment out the seadas.config file?

Though, after starting server and clicking "OCSSW", the error massage (Remote server is down. OCSSW is not accessible.
Please start OCSSW remote server) is not showed up :D

But I cant go through the OCSSW options ( I mean drop-down after clicking ocssw). I have figured out that I skipped the install of
install_ocssw.py .

Question 2:

For install_ocssw.py, Should I follow the Manual Installation of (https://seadas.gsfc.nasa.gov/downloads/) ?

or I have to install SeaDAS Visualization Installers and/or Install with GUI or Manual Installation?

Question 3:

I have tried to install intall_ocssw.py by running two commands (as below)
  $ chmod +x install_ocssw.py
./install_ocssw.py --install-dir=/mnt/c/Users/antu/seadasClientServerShared/ocssw --git-branch=v2.17.1  --aqua --seawifs

After executing this command, this type error (Below) showed up and also frequently starting and closing download of 1.2G data to my selected loacation.
"CError - Executing command "cd /mnt/c/Users/antu/seadasClientServerShared/ocssw; wget --tries=5 --wait=5 --continue https://oceandata.sci.gsfc.nasa.gov/ocssw/common.bundle"
Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '/home/antu/.wget-hsts'. HSTS will be disabled."
"Installing common (2 of 12)
Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '/home/antu/.wget-hsts'. HSTS will be disabled."

I also tried to run this code in home directory like:
./install_ocssw.py --install-dir=$HOME/ocssw --git-branch=v2.17.1 --aqua --seawifs

same type of errors showed up (Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '/home/antu/.wget-hsts'. HSTS will be disabled.
Error - Executing command "cd /home/antu/ocssw; wget --tries=5 --wait=5 --continue https://oceandata.sci.gsfc.nasa.gov/ocssw/common.bundle"
Will not apply HSTS. The HSTS database must be a regular and non-world-writable file.
ERROR: could not open HSTS store at '/home/antu/.wget-hsts'. HSTS will be disabled.)

Question
How I can solve this problem and install ocssw.py?

Thanks Antu

[OB.DAACx - SeanBailey]

Try:

seadas.config
seadas.ocssw.sharedDir=C:\\Users\\antu\\seadasClientServerShared
seadas.client.id=antu

You will get the "remote server is down" message unless you start the server before the SeaDAS GUI.
You do that in the server terminal window via:
java -Xmx4G -jar seadas-ocsswserver.jar ocsswserver.config


With the server running (and properly configured to connect to the client), you can use the GUI to install OCSSW.

The manual install command would be:
install_ocssw.py --install-dir=$HOME/ocssw --git-branch=v7.5  --aqua --seawifs 

"--git-branch=v2.17.1" will not work

The wget "HSTS" errors suggest you have permission issues.  I can't help with that...

[antu]

Thank you very much SeanBailey.
I am feeling very silly for mistyping the config code. Now server starts and I can install ocssw with GUI. But I am facing another problem. After download data for a long time, I have found such type of error massages:

Checking out files:  92% (115/124)  
Checking out files:  93% (116/124)  
Checking out files:  94% (117/124)  
Checking out files:  95% (118/124)  
Checking out files:  95% (119/124)  
Checking out files:  96% (120/124)  
Checking out files:  97% (121/124)  
Checking out files:  98% (122/124)  
Checking out files:  99% (123/124)  
Checking out files: 100% (124/124)  
Checking out files: 100% (124/124), done.
fatal: unable to access 'https://oceandata.sci.gsfc.nasa.gov/ocssw/common.git/': gnutls_handshake() failed: The TLS connection was non-properly terminated.
Error - Executing command "cd /mnt/c/Users/antu/seadasClientServerShared/ocssw; wget --tries=5 --wait=5 --continue https://oceandata.sci.gsfc.nasa.gov/ocssw/common.bundle"
Error - Could not run "cd /mnt/c/Users/antu/seadasClientServerShared/ocssw/share/common; git fetch -q > /dev/null"

How can I solve this problem and install ocssw?

I thought, it was problem in my git. So, I was also trying to install git according to this instruction (https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=27714). But there also I have shown some error massage:

antu@DESKTOP-7K5SEI6:~$ which git
/usr/bin/git
antu@DESKTOP-7K5SEI6:~$ sudo apt-get install build-essential libcurl4-openssl-dev
[sudo] password for antu:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  dpkg-dev fakeroot g++ g++-7 libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libdpkg-perl libfakeroot libfile-fcntllock-perl libstdc++-7-dev
  make
Suggested packages:
  debian-keyring g++-multilib g++-7-multilib gcc-7-doc libstdc++6-7-dbg libcurl4-doc libidn11-dev libkrb5-dev libldap2-dev librtmp-dev libssh2-1-dev libssl-dev
  pkg-config zlib1g-dev bzr libstdc++-7-doc make-doc
The following NEW packages will be installed:
  build-essential dpkg-dev fakeroot g++ g++-7 libalgorithm-diff-perl
  libalgorithm-diff-xs-perl libalgorithm-merge-perl
  libcurl4-openssl-dev libdpkg-perl libfakeroot libfile-fcntllock-perl
  libstdc++-7-dev make
0 upgraded, 14 newly installed, 0 to remove and 33 not upgraded.
Need to get 9549 kB/10.5 MB of archives.
After this operation, 47.3 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libstdc++-7-dev amd64 7.4.0-1ubuntu1~18.04 [1467 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 g++-7 amd64 7.4.0-1ubuntu1~18.04 [7575 kB]
Err:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 g++-7 amd64 7.4.0-1ubuntu1~18.04
  Connection failed [IP: xx.xxx.xx.xx 80]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 g++ amd64 4:7.4.0-1ubuntu2.2 [1568 B]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libdpkg-perl all 1.19.0.5ubuntu2.1 [211 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcurl4-openssl-dev amd64 7.58.0-2ubuntu3.6 [295 kB]
Fetched 1786 kB in 5min 39s (5268 B/s)
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/g/gcc-7/g++-7_7.4.0-1ubuntu1~18.04_amd64.deb  Connection failed [IP: xx.xxx.xx.xx 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
antu@DESKTOP-7K5SEI6:~$ apt-get install git
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?

What should I do now?

[gnwiii]

[url=][/url]As you have discovered, SeaDAS is a large and complicated system and can take some effort to install, especially if your internet connection is not robust.  I have often found that internet access becomes erratic from home in the evening (presumably many households are streaming video).   Two things you can do are a) find a time when the network is less congested, and b) download big files manually using the "resume partial download" option in either curl or wget.   If you continue to have problems with either curl or wget, try the other program (note that the options are different) as there are differences in the way the two programs handle erratic connections.   This may seem excessively complicated, but learning to deal with less-than-ideal internet connections is a useful skill to have.

When you are working through a complex configuration process it is very helpful to have someone else double check your work to catch mistakes right away.  For workshops on Ocean Remote Sensing it is very effective to use a "buddy system" pairing up the participants to install software on half the computers at a time.  In these workshops it is also useful to spend an afternoon introducing linux command-line concepts.  Maybe you can find a "buddy" who is already familiar with linux.  Many areas have linux users groups whose members will be more than happy to help you.  There are a number of excellent on-line resources where you can learn basics of the WSL command-line.  A good example is
Linux Command.

1. You asked: "How can I solve this problem and install ocssw?"

The install_ocssw.py script can be run again after a encountering a problem and will usually get a bit further with each attempt, but it may be more efficient to use the script with the "local" install option after downloading the git bundles one-by-one using curl or wget.   This must be done using the WSL bash window (which will be easier once you are comfortable with the command-line).

You should check that your WSL linux installation has the required versions of git, curl, and wget.  To determine which versions you have, run:

$ git --version; curl --version | head -1; wget --version|head -1
git version 2.17.1
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.0g zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
GNU Wget 1.19.4 built on linux-gnu.
(note that this is for recently updated WSL Ubuntu 18.04, so may not match your system).

2. You asked "What should I do now?" in reference to:

antu@DESKTOP-7K5SEI6:~$ apt-get install git
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)


Because apt-get may modify the linux "system", it must be run with elevated privileges, e.g., using "sudo apt-get install git".

[antu]

Thank you very much gnwiii.
I am trying to install ocssw with GUI. As my internet speed is not so good , it is taking time.
If I am failed to download it, I will go for manual installation as you said. Thanks again for give explanations for all of my problems.
Regards,
Antu

[antu]

Thank you very much Mr. gnwiii and Mr. SeanBailey. I have successfully install ocssw.
Though I am facing some problem in L3Bin. I like to create a new post for it. I hope we will solve this problem too.

Greeting.