Earthdata Forum

I am using wget to get all file under a directory, but it returned nothing. The script i used as follows:
============================================================
URL=https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
USERNAME=myuserid
PASSWORD=mypassword

cd ~
touch .netrc
echo "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" > .netrc
chmod 0600 .netrc
touch .urs_cookies
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition --recursive --wait=60 --no-parent --reject "index.html*" --execute robots=off $URL
~


==============================================================================


------------------------------------------------------------------------
the following is the returned message on screen.
------------------------------------------------------------------------------------------------------------------------------
--2021-03-04 21:34:31-- https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
Resolving asdc.larc.nasa.gov (asdc.larc.nasa.gov)... 198.119.164.126, 2001:4d0:2340:4001::7e
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?splash=false&client_id=iQGRa5KtDl_e-fgYqB5x5Q&response_type=code&redirect_uri=https%3A%2F%2Fasdc.larc.nasa.gov%2Fdata%2Furs&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8 [following]
--2021-03-04 21:35:31-- https://urs.earthdata.nasa.gov/oauth/authorize?splash=false&client_id=iQGRa5KtDl_e-fgYqB5x5Q&response_type=code&redirect_uri=https%3A%2F%2Fasdc.larc.nasa.gov%2Fdata%2Furs&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://asdc.larc.nasa.gov/data/urs?code=efdd497faef09abeba1758558518ea32a9b6a47a2a54b8a88c9e5d9b12391ebf&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8 [following]
--2021-03-04 21:36:31-- https://asdc.larc.nasa.gov/data/urs?code=efdd497faef09abeba1758558518ea32a9b6a47a2a54b8a88c9e5d9b12391ebf&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/ [following]
--2021-03-04 21:37:31-- http://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/ [following]
--2021-03-04 21:38:31-- https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/index.html’

[ <=> ] 384,362 754KB/s in 0.5s

2021-03-04 21:38:32 (754 KB/s) - ‘asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/index.html’ saved [384362]

Removing asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/index.html since it should be rejected.

FINISHED --2021-03-04 21:38:32--
Total wall clock time: 4m 1s
Downloaded: 1 files, 375K in 0.5s (754 KB/s)

Dear Pubu Ciren,

After reaching out to our development team I received the following guidance:
Below I’ve a script that worked for me. If that does not work, could you send me a copy of the file that it downloads?
Additionally, please verify that your earthdata login works by logging into the following site: Earthdata Login (nasa.gov)

$ URL=https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Prov-V3-40/2020/05/CAL_LID_L2_05kmAPro-Prov-V3-40.2020-05-01T01-49-51ZD.hdf
$ USERNAME=njester
$ PASSWORD=<my password>
$ cd ~
$ touch .netrc
$ echo "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" > .netrc
$ chmod 0600 .netrc
$ touch .urs_cookies

Then one of the two methods:
$ wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies $URL
or
$ curl -b ~/.urs_cookies -c ~/.urs_cookies --remote-name --___location --netrc $URL

If you continue to have difficulties, please let us know.

Also, it looks like the link to earthdata login didn't make it into the above post, but since you were able to log into the forum, your Earthdata login should be fine.

Thanks! it works for one file, but if I would like to download all file under one directory, what should do?

I used to be able to use option "-A hdf" in wget, but it did not work any more..

Do you have any suggestion?


thanks!



-pubu