downloading CALIPSO data from one directory

[pciren]

I am using wget to get all file under a directory, but it returned nothing. The script i used as follows:
============================================================
URL=https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
USERNAME=myuserid
PASSWORD=mypassword

cd ~
touch .netrc
echo "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" > .netrc
chmod 0600 .netrc
touch .urs_cookies
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition --recursive --wait=60 --no-parent --reject "index.html*" --execute robots=off $URL
~


==============================================================================


------------------------------------------------------------------------
the following is the returned message on screen.
------------------------------------------------------------------------------------------------------------------------------
--2021-03-04 21:34:31-- https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
Resolving asdc.larc.nasa.gov (asdc.larc.nasa.gov)... 198.119.164.126, 2001:4d0:2340:4001::7e
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?splash=false&client_id=iQGRa5KtDl_e-fgYqB5x5Q&response_type=code&redirect_uri=https%3A%2F%2Fasdc.larc.nasa.gov%2Fdata%2Furs&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8 [following]
--2021-03-04 21:35:31-- https://urs.earthdata.nasa.gov/oauth/authorize?splash=false&client_id=iQGRa5KtDl_e-fgYqB5x5Q&response_type=code&redirect_uri=https%3A%2F%2Fasdc.larc.nasa.gov%2Fdata%2Furs&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://asdc.larc.nasa.gov/data/urs?code=efdd497faef09abeba1758558518ea32a9b6a47a2a54b8a88c9e5d9b12391ebf&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8 [following]
--2021-03-04 21:36:31-- https://asdc.larc.nasa.gov/data/urs?code=efdd497faef09abeba1758558518ea32a9b6a47a2a54b8a88c9e5d9b12391ebf&state=aHR0cDovL2FzZGMubGFyYy5uYXNhLmdvdi9kYXRhL0NBTElQU08vTElEX0wyXzA1a21BTGF5LVByb3YtVjMtNDEvMjAyMC8xMi8
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/ [following]
--2021-03-04 21:37:31-- http://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/ [following]
--2021-03-04 21:38:31-- https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/index.html’

[ <=> ] 384,362 754KB/s in 0.5s

2021-03-04 21:38:32 (754 KB/s) - ‘asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/index.html’ saved [384362]

Removing asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmALay-Prov-V3-41/2020/12/index.html since it should be rejected.

FINISHED --2021-03-04 21:38:32--
Total wall clock time: 4m 1s
Downloaded: 1 files, 375K in 0.5s (754 KB/s)

[ASDC - rbeck]

Dear Pubu Ciren,

After reaching out to our development team I received the following guidance:
Below I’ve a script that worked for me. If that does not work, could you send me a copy of the file that it downloads?
Additionally, please verify that your earthdata login works by logging into the following site: Earthdata Login (nasa.gov)

$ URL=https://asdc.larc.nasa.gov/data/CALIPSO/LID_L2_05kmAPro-Prov-V3-40/2020/05/CAL_LID_L2_05kmAPro-Prov-V3-40.2020-05-01T01-49-51ZD.hdf
$ USERNAME=njester
$ PASSWORD=<my password>
$ cd ~
$ touch .netrc
$ echo "machine urs.earthdata.nasa.gov login $USERNAME password $PASSWORD" > .netrc
$ chmod 0600 .netrc
$ touch .urs_cookies

Then one of the two methods:
$ wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies $URL
or
$ curl -b ~/.urs_cookies -c ~/.urs_cookies --remote-name --___location --netrc $URL

If you continue to have difficulties, please let us know.

[njester]

Also, it looks like the link to earthdata login didn't make it into the above post, but since you were able to log into the forum, your Earthdata login should be fine.

[pciren]

Thanks! it works for one file, but if I would like to download all file under one directory, what should do?

I used to be able to use option "-A hdf" in wget, but it did not work any more..

Do you have any suggestion?


thanks!



-pubu