wget EarthData Login does not work

[bruce]

bcb@modis:/v/aqua_data/gom$ python -V
Python 2.7.12

ubuntu 16.04 LTS

bcb@modis:/v/aqua_data/gom$ update_luts.py -v aqua

Downloading files into /seadas/seadas-7.5.3/ocssw/var/common
...no new files.

Downloading files into /seadas/seadas-7.5.3/ocssw/var/modis
...no new files.

Downloading files into /seadas/seadas-7.5.3/ocssw/var/modisa
...no new files.

Downloading files into /seadas/seadas-7.5.3/ocssw/var/modisa/cal/OPER
...no new files.

Downloading files into /seadas/seadas-7.5.3/ocssw/var/modisa/xcal/OPER
...no new files.

bcb@modis:/v/aqua_data/gom$ getanc.py -v A2020013173000.L1A_LAC
Searching database: /seadas/seadas-7.5.3/ocssw/var/ancillary_data.db
Aqua
()
Input file: A2020013173000.L1A_LAC
Sensor    : Aqua
Start time: 2020013173000
End time  : 2020013173500
()
*** WARNING: No optimal SST files found.
*** WARNING: No optimal ICE files found.
()
Created 'A2020013173000.L1A_LAC.anc' l2gen parameter text file:

()
*** WARNING: The following ancillary data types were missing or are not optimal:  MET OZONE SST Sea Ice
*** Beware that certain MET and OZONE files just chosen by this program are not optimal.
*** For near real-time processing the remaining files may become available soon.

the .anc file is empty.

[OB.DAACx - SeanBailey]

The issue is indeed with the python download function.  We're working on an update to get it to play nice with the new authentication requirement.
In the meantime, you might try using the -d (--disable-download) option to get the list of files then pull them down using wget or cURL...
not ideal, but should work...

The update_luts.py works because it's pulling data that are not part of the authentication requirement.

Sean

[bbbarnes]

One (hopefully helpful) note about cURL downloads:
The "data downloads methods" page (https://oceancolor.gsfc.nasa.gov/data/download_methods/) says that "cURL has no method for downloading a list of URLs (although it can download multiple URLs on the command line)." While the page suggests a loop to accomplish this, with these new authentication methods, each new connection takes a while. Thus, a much better way is to use sed within a curl command on the command line.

For example:

>curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n --remote-name-all https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/{$(sed ':a;N;$!ba;s/\n/,/g' /path/to/list_of_files.txt)}

This will create one connection to NASA through which all files will be downloaded. The /path/to/list_of_files.txt in this instance needs to have the granule names, with each granule listed on a new line, e.g.,:

>cat /path/to/list_of_files.txt

A2020005000500.L1A_LAC.bz2
A2020005001000.L1A_LAC.bz2
...

-brian

[gnwiii]

Did you try:
$ wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1EPHND.P2020013.1200.003

or
$ curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1EPHND.P2020013.1200.003

MacOS El Capitan with macports Wget 1.20.3 working from behind an enterprise firewall gave:
--2020-01-16 11:21:09--  https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1EPHND.P2020013.1200.003
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: /ob/getfile/PM1EPHND.P2020013.1200.003 [following]
--2020-01-16 11:21:09--  https://oceandata.sci.gsfc.nasa.gov/ob/getfile/PM1EPHND.P2020013.1200.003
Reusing existing connection to oceandata.sci.gsfc.nasa.gov:443.
HTTP request sent, awaiting response... 200 OK
Length: 5531136 (5.3M) [application/octet-stream]
Saving to: ‘PM1EPHND.P2020013.1200.003’

PM1EPHND.P2020013.1200.003    100%[=================================================>]   5.27M  7.91MB/s    in 0.7s

2020-01-16 11:21:10 (7.91 MB/s) - ‘PM1EPHND.P2020013.1200.003’ saved [5531136/5531136]

Wget also worked with --auth-no-challenge=off.  Note that man wget says  --auth-no-challenge=on is "not recommended, and is intended only to support some few obscure servers".  

On the same system with macports curl  7.68.0 (x86_64-apple-darwin15.6.0) libcurl/7.68.0 OpenSSL/1.1.1d zlib/1.2.11 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.1.1)
gave:
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 5401k  100 5401k    0     0  2808k      0  0:00:01  0:00:01 --:--:-- 6507k

[dem1]

Hi,

I managed to update our Python download chain based on the Python Requests module with the help of this doc:
https://wiki.earthdata.nasa.gov/display/EL/How+To+Access+Data+With+Python (second example)
which is also implemented in this project:
https://github.com/DHI-GRAS/earthdata-download/blob/master/earthdata_download/download.py (implemented by the EarthdataSession class)
Usage example:
session = EarthdataSession(user, password) # This session will be used for all following downloads
product_name = 'T2019365193500.L2_LAC_OC.nc'
response = session.request('GET', 'https://oceandata.sci.gsfc.nasa.gov/cgi/getfile/' + product_name)
response.raise_for_status()
with open(product_name, 'wb') as f:
        f.write(response.content)

This method needn't .netrc and .urs_cookies files.
The first download is slow (~ a minute), I think because authentication, but the following downloads are a lot faster (~ 10x faster).

Julien

[monger]

Hi -- From this thread on EarthData Login/Downloads I see that Sean recognized (a ways down in the thread) the getanc.py is not working well with the new authentication requirements for ocssw data.  It is also not working for me. I tired using getanc.py from the unix command line on an extaracted viirs snap l1a file using seadas 7.5.3 (on MacOS Catalina).   I was wondering how the update might occur? Would I download a new version of getanc.py file from some website or will we use the Update Processors feature in seadas?

Thanks for any advice!
Bruce

[OB.DAACx - SeanBailey]

The update will most likely go out as a normal SeaDAS OCSSW update, so the GUI Update Processors or a manual install_ocssw call.

I'll warn you now...the update will impose a requirement for python 3...

Sean

[gnwiii]

If you have a very long list then xargs command < list can be used.  From the man page:

       The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options
       are  used).   The  specified  command  will  be invoked as many times as necessary to use up the list of input
       items.  In general, there will be many fewer invocations of command than there were items in the input.   This
       will  normally have significant performance benefits.

[ionutserban]

Hello.
I am trying to use modis_GEO.py on MODIS Aqua L1A product.

As others mentioned already I receive the following error. I understood that there are issues caused by the new requirements
for EarthData Login for data downloads.
The command I use is: modis_GEO.py A2011002110500.L1A_LAC
I got:

URL attempted: oceandata.sci.gsfc.nasa.gov
Well, this is embarrassing...an error occurred that we just cannot get past...
Here is what we know: ('The read operation timed out',)
Please retry this request at a later time.
*** ERROR: The HTTP transfer failed with status code 302.
*** Please check your network connection and for the existence of the remote file:
*** oceandata.sci.gsfc.nasa.gov/cgi/getfile/PM1EPHND.P2011001.1200.003
***
*** Also check to make sure you have write permissions under the directory:
*** /home/ionut/seadas-7.5.3/ocssw/var/anc/2011/001
()
Cannot create geolocation from A2011002110500.L1A_LAC; exiting.

My question is:
Is there I way to manually download the required files (using the new download methods) and then run the modis_GEO.py? 
Are there any other suggestions on how can I solve this?

Thank you for your help!
Best,
Ionut

[bruce]

There are a couple of threads in the seadas 7 forum that address a solution to this problem.