wget syntax to download a whole directory

[rpalikonda]

Hi,
Using Direct Data Download, I want to use wget to get all the files and subdirectory. For e.g
I want to get all the subdirectories and files under 2002.

https://asdc.larc.nasa.gov/data/CERES/G ... _V01/2002/

If sub-folders is not possible then all the files under the directory

https://asdc.larc.nasa.gov/data/CERES/G ... 1/2002/001

Thanks
Rabi

[joseph.f.koch]

Hi Rabi,

Here are some instructions for downloading ASDC data using Wget:

viewtopic.php?f=7&t=714&p=1565&hilit=Da ... 289a#p1565

You may also find the 'Scripts for Downloading Data' useful which contains instructions as well. You can find these scripts on the main webpage of our Direct Data Download site, in the upper right hand corner https://asdc.larc.nasa.gov/data/

Please let us know if you have any additional questions or need further assistance.

Thanks,
Joe Koch
NASA Langley ASDC User Services

[rpalikonda]

Hi Joe,

I followed the directions in the ASDC direct download link. Created the .netrc, included the line with username and passwd. When I execute the following cmd below is a html file called CER_GEO_Ed4_GOE08_NH_V01_2002.001.0045.08K.nc

that is 12731 bytes. What am I doing wrong. Thx for your help.

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies https://asdc.larc.nasa.gov/data/CERES/G ... 045.08K.nc

--2021-04-13 14:32:31-- https://asdc.larc.nasa.gov/data/CERES/G ... 045.08K.nc
Resolving asdc.larc.nasa.gov (asdc.larc.nasa.gov)... 198.119.164.126
Connecting to asdc.larc.nasa.gov (asdc.larc.nasa.gov)|198.119.164.126|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/au ... UuMDhLLm5j [following]
--2021-04-13 14:32:31-- https://urs.earthdata.nasa.gov/oauth/au ... UuMDhLLm5j
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: â

CER_GEO_Ed4_GOE08_NH_V01_2002.001. [ <=> ] 12.43K --.-KB/s in 0.02s

2021-04-13 14:32:31 (667 KB/s) - â saved [12731]

[joseph.f.koch]

Hi Rabi,

Please try replacing your Wget line with the following command:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition $URL

Hope this helps, and please let us know if you run into any further issues with accessing your files.

Thanks,
Joe Koch
NASA Langley ASDC User Services

[rpalikonda]

Joe,

Thanks!. That worked to download a single file. If I wanted to get the all the files under "001" directory what would the syntax be.I tried the '-r' option with no luck.
Rabi

[aseifert]

Hi,

I had similar problems with CERES data and was lucky enough to find your thread.
This works for me for a whole directory of FlashFlux SSF:

wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition -r -np -nc -nH --cut-dirs=4 -e robots=off https://asdc.larc.nasa.gov/data/CERES/F ... A/2021/01/

Best regards, Axel