Slow download speeds for MISR data

[joeyhotz]

Hi all, I have been trying to use a script which I made in R to download MISR NetCDF files from the MIL2ASAE.003 dataset on the OPeNDAP server.

In my experience, I have found that the download speeds for these NetCDF files are quite quick at first, but after a certain point (around 50 files), it starts to take around 10 minutes to download a single NetCDF file.

I am downloading approximately 250 files for each year (from 2000-2021) in order to make annual datasets, and as it stands, it takes around 48 hours to download all the necessary NetCDF files to create each of these individual annual datasets.

Does anyone know of a workaround to make these file downloads quicker? I suspect that there might be some system which times me out after too many download requests in a short time period, but I have not been able to find any information on whether this is the case.

[ASDC - ingridgs]

Hello,

You can try to download data via Direct Data Download:
https://asdc.larc.nasa.gov/data/MISR/

Also, you can find here in the Forum scripts to download multiple files with Python and wget.
wget: viewtopic.php?t=2329
Python: viewtopic.php?t=2330

Regards,

[joeyhotz]

Thanks for the reply, I'll try and edit my code to use wget in the script.

Do you know if using wget would help alleviate the issue which I have been having with slow downloads? I have been using the "download.file()" function in R instead of wget and I have noticed that it often takes around 10 minutes to download a single NetCDF file when running my script.

[ASDC - ingridgs]

Hi @joeyhotz

I have not used R to download multiple files from ASDC data, so I wouldn't know how to answer. However, I have tried both the wget and Python scripts, and I have always been able to retrieve multiple files within minutes.

Regards,