Earthdata Forum

Welcome to the Earthdata Forum! Here, the scientific user community and subject matter experts from NASA Distributed Active Archive Centers (DAACs), and other contributors, discuss research needs, data, and data applications.
https://forum.earthdata.nasa.gov/

Non-interactive downlaods through Earthdata login

https://forum.earthdata.nasa.gov/viewtopic.php?t=1807

Page 1 of 1

Non-interactive downlaods through Earthdata login

Posted: Wed Nov 20, 2019 4:36 pm America/New_York
by b.mccabe
With regards to the recently announced move the requiring EarthData login for data downloads, we have been unable to make non-interactive downloads work using the guidance at https://oceancolor.gsfc.nasa.gov/data/download_methods/.  Using a HICO file as a test, as it is already requires EarthData login, attempting to download the file results in the source code for the login page being retrieved instead:

[bmccabe1@sora ~]$ wget --user=b.mccabe --password='mypassword' https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc
--2019-11-20 21:17:41--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Nwx7SywSELFPjkOk6Ifsjw&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict [following]
--2019-11-20 21:17:42--  https://urs.earthdata.nasa.gov/oauth/authorize?client_id=Nwx7SywSELFPjkOk6Ifsjw&response_type=code&redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘H2014255140230.L2_ISS_OC.nc’

    [ <=>                                                                                                                                      ] 12,663      --.-K/s   in 0.001s 

2019-11-20 21:17:42 (14.9 MB/s) - ‘H2014255140230.L2_ISS_OC.nc’ saved [12663]

[bmccabe1@sora ~]$ head -n 10 H2014255140230.L2_ISS_OC.nc
<!DOCTYPE html>
<!--[if lt IE 7]><html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]><html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]><html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--><html lang="en" class="no-js"><!--<![endif]-->
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
    <title>Earthdata Login</title>
    <meta name="description" content="Earthdata Login">

Attempting to store my credentials via the ~/.netrc file and calling wget with a cookie fares no better:

[bmccabe1@sora ~]$ rm H2014255140230.L2_ISS_OC.nc
[bmccabe1@sora ~]$ echo 'machine urs.earthdata.nasa.gov login b.mccabe password mypassword' > ~/.netrc
[bmccabe1@sora ~]$ chmod 0600 ~/.netrc
[bmccabe1@sora ~]$ touch ~/.urs_cookies
[bmccabe1@sora ~]$ wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc
--2019-11-20 21:23:32--  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc
Resolving oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:2418:128::84
Connecting to oceandata.sci.gsfc.nasa.gov (oceandata.sci.gsfc.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict&response_type=code&client_id=Nwx7SywSELFPjkOk6Ifsjw [following]
--2019-11-20 21:23:32--  https://urs.earthdata.nasa.gov/oauth/authorize?redirect_uri=https%3A%2F%2Foceandata.sci.gsfc.nasa.gov%2Fhico%2Fgetfile%2Frestrict&response_type=code&client_id=Nwx7SywSELFPjkOk6Ifsjw
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... xx.xxx.xx.xx, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|xx.xxx.xx.xx|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘H2014255140230.L2_ISS_OC.nc’

    [ <=>                                                                                                                                      ] 12,663      --.-K/s   in 0.001s 

2019-11-20 21:23:32 (14.6 MB/s) - ‘H2014255140230.L2_ISS_OC.nc’ saved [12663]

[bmccabe1@sora ~]$ head -n 10 H2014255140230.L2_ISS_OC.nc
<!DOCTYPE html>
<!--[if lt IE 7]><html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
<!--[if IE 7]><html class="no-js lt-ie9 lt-ie8"> <![endif]-->
<!--[if IE 8]><html class="no-js lt-ie9"> <![endif]-->
<!--[if gt IE 8]><!--><html lang="en" class="no-js"><!--<![endif]-->
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
    <title>Earthdata Login</title>
    <meta name="description" content="Earthdata Login">

Using curl yields similar results.  I am able to successfully download HICO files using the web browser.  Any ideas why I am unable to successfully download a file with Earthdata login non-interactively?

Thanks,
Benjamin McCabe/Computer Scientist, Direct Readout Laboratory
NASA/Goddard Space Flight Center
http://directreadout.sci.gsfc.nasa.gov/
https://www.facebook.com/directreadout/
https://twitter.com/NASA_DRL

Non-interactive downlaods through Earthdata login

Posted: Tue Dec 03, 2019 3:45 pm America/New_York
by b.mccabe
Any ideas why our download attempts are resulting in us downloading the login page rather than the target file?  We very much would like to have our applications updated to be able to continue retrieving the ancillary files we need before this change goes into effect.

Thanks,
Benjamin McCabe/Computer Scientist, Direct Readout Laboratory
NASA/Goddard Space Flight Center
http://directreadout.sci.gsfc.nasa.gov/
https://www.facebook.com/directreadout/
https://twitter.com/NASA_DRL

Non-interactive downlaods through Earthdata login

Posted: Tue Dec 03, 2019 6:03 pm America/New_York
by OB.DAACx - SeanBailey
Hmmm...wget gives me the same problem, but curl works:

curl -O -b ~/.urs_cookies -c ~/.urs_cookies -L -n https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc

We'll dig into why the wget behavior changed (it worked for me when we wrote the original tutorial several year ago).  In the meantime, give curl a try and see if that will work for you.

Sean

Non-interactive downlaods through Earthdata login

Posted: Wed Dec 04, 2019 2:22 pm America/New_York
by b.mccabe
Thanks.  I was able to perform a successful download using curl, and look forward to hearing your findings regarding wget.

Thanks,
Benjamin McCabe/Computer Scientist, Direct Readout Laboratory
NASA/Goddard Space Flight Center
http://directreadout.sci.gsfc.nasa.gov/
https://www.facebook.com/directreadout/
https://twitter.com/NASA_DRL

Non-interactive downlaods through Earthdata login

Posted: Fri Dec 06, 2019 2:36 pm America/New_York
by OB.DAACx - SeanBailey
Benjamin,
Might have been a wget version problem - seems there was an issue with .netrc implementation pre 1.19.1

Using wget versions 1.19.4 (Ubuntu 18.04 Linux) and 1.20.3 (Mac), the following worked for me:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --no-check-certificate https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc

...yes pretty much the previous example.

Sean

Non-interactive downlaods through Earthdata login

Posted: Tue Dec 10, 2019 2:52 pm America/New_York
by b.mccabe
Perhaps, but unfortunately wget 1.14 is the latest available through the CentOS 7 package repository. Also, I note that my first command:
wget --user=b.mccabe --password='mypassword' https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc
Does not work despite not using .netrc.

Thanks,
Ben

Non-interactive downlaods through Earthdata login

Posted: Tue Dec 10, 2019 3:46 pm America/New_York
by OB ODPS - towens
Can you build wget-1.20 from source?  https://ftp.gnu.org/gnu/wget/

Tommy

Non-interactive downlaods through Earthdata login

Posted: Mon Dec 16, 2019 3:39 pm America/New_York
by b.mccabe
We would rather not move away from the package manager managed version of the software on this system if at all possible, especially since we have other applications which are successfully using the existing wget version which could potentially be impacted by an upgrade.

Thanks,
Ben

Non-interactive downlaods through Earthdata login

Posted: Mon Dec 16, 2019 7:01 pm America/New_York
by OB.DAACx - SeanBailey
Ben,

Understandable. 

Try adding --auth-no-challenge to the command line options for wget.  I found a machine with 1.14 on it and that seemed to do the trick, e.g.:
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --auth-no-challenge  https://oceandata.sci.gsfc.nasa.gov/hico/getfile/H2014255140230.L2_ISS_OC.nc

...or use cURL...that also worked for me on that Centos7 machine :grin:

Sean
All times are UTC-04:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited