VIIRS&MODIS l2 product fetch - close down of ftp from samoa

[adybbroe]

Hello,

We have been fetching VIIRS and MODIS ocean color products via an ftp account for some years. This fetch stopped working 21/2, and I see in the announcements that this was planned. However, the earliest announcement is from 28/2-2018. But probably I have missed earlier ones?

Anyhow, we did a fetch like this:

lftp -c "open ftp://samoa.gsfc.nasa.gov && mirror -v -c --newer-than=now-1day <subscription catalougue> <local directory>"

Can you give us some guidance on how to migrate this, so we can get up and running again?

Thanks you!
Adam

[OB.DAACx - SeanBailey]

Adam,
Have you tried to manually connect to samoa? 

The samoa server is still up and serving data.  An announcement about the new method of subscription access will be going out shortly and both methods will be available until 19 March.  On the 19th, samoa will be decommissioned as an FTP server and only the new method will work. 

Sean

[adybbroe]

Sean,

Thanks.

Yes, we have tried manually on different machines. Like this:
lftp -c "open ftp://samoa.gsfc.nasa.gov && mirror -v -c --newer-than=now-1day /subscriptions/MODIST/XM/martin.hansson/2113/ /data/prod/satellit/modis/lvl2"

It doesn't respond at all.

And same in a (Firefox) browser.
-Adam

[OB.DAACx - SeanBailey]

I meant, something like "ftp samoa.gsfc.nasa.gov"....

but as FTP will be moot in a week - try the new method:
https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=35661#pid35661

For your subscription #2113:
wget --post-data="subID=2113&subType=2&addurl=1&results_as_file=1" -O - https://oceandata.sci.gsfc.nasa.gov/api/file_search

Unfortunately, we didn't implement the staging for the new method until last week, so to get the data you may have missed you'll need
access to samoa (before it becomes a dodo).  Send an email to connection_problems@oceancolor.sci.gsfc.nasa.gov  with the information
requested in this post:
https://oceancolor.gsfc.nasa.gov/forum/oceancolor/topic_show.pl?pid=27677

Sean

[adybbroe]

Many thanks for the help Sean!

We manage to get data now. However, we see that the hdf and netCDF files are packaged in gzipped tar files. See below. This is not a major problem, but of course requires an extra step on the reception side before delivering the files to our users. I believe the data was not put in tar files before, right?

-Adam


lx <a000680>:/tmp/nasa_ocean_color % ll
total 316116
-rw-rw-r--. 1 a000680 users   2626755 Mar 14 03:54 T2018072175500.L1A_LAC_S2113.tgz
-rw-rw-r--. 1 a000680 users   1809678 Mar 13 23:04 T2018072130500.L1A_LAC_S2113.tgz
-rw-rw-r--. 1 a000680 users     63108 Mar 13 21:23 T2018072095500.L1B_LAC.S2113.hdf.BAWS_NewBaltic.TC.png
-rw-rw-r--. 1 a000680 users  32571722 Mar 13 21:23 T2018072095500.L1A_LAC_S2113.tgz
-rw-rw-r--. 1 a000680 users    194039 Mar 13 20:39 T2018072113000.L1B_LAC.S2113.hdf.BAWS_NewBaltic.TC.png
-rw-rw-r--. 1 a000680 users  98981810 Mar 13 20:39 T2018072113000.L1A_LAC_S2113.tgz
-rw-rw-r--. 1 a000680 users     52446 Mar 13 19:59 T2018072081500.L1B_LAC.S2113.hdf.BAWS_NewBaltic.TC.png
-rw-rw-r--. 1 a000680 users  10746132 Mar 13 19:59 T2018072081500.L1A_LAC_S2113.tgz
-rw-rw-r--. 1 a000680 users     41665 Mar 13 19:55 T2018072112500.L1B_LAC.S2113.hdf.BAWS_NewBaltic.TC.png
-rw-rw-r--. 1 a000680 users   9318551 Mar 13 19:55 T2018072112500.L1A_LAC_S2113.tgz
-rw-rw-r--. 1 a000680 users     43064 Mar 13 19:52 T2018072081000.L1B_LAC.S2113.hdf.BAWS_NewBaltic.TC.png
-rw-rw-r--. 1 a000680 users    297187 Mar 13 18:07 T2018072095000.L1B_LAC.S2113.hdf.BAWS_NewBaltic.TC.png
-rw-rw-r--. 1 a000680 users 166925032 Mar 13 18:07 T2018072095000.L1A_LAC_S2113.tgz
lx <a000680>:/tmp/nasa_ocean_color % tar tvzf T2018072175500.L1A_LAC_S2113.tgz
-rw-rw-r-- sdpsoper/sdpsoper 13926559 2018-03-14 03:52 T2018072175500.L1A_LAC.S2113_BAWS_NewBaltic.hdf
-rw-rw-r-- sdpsoper/sdpsoper   400792 2018-03-14 03:53 ./T2018072175500.L2_LAC.S2113_BAWS_NewBaltic.nc


[OB.DAACx - SeanBailey]

Adam,

Correct, to reduce the number of files we need to manage and users need to download, we are now creating tar files if the request produces more than a single file.

Regards,
Sean

[adybbroe]

Sean,

Thanks, sounds reasonable.
I think we only need the netCDF file, so we will try change our subscription then, to make it even more simple for us, and save bandwidth.

Cheers
-Adam